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Abstract 

This  annotated  bibliography  is  a  collection  of  relevant  literature  for  researchers,  designers,  and 
developers  of  advanced  Interactive  Electronic  Technical  Manuals  (lETMs).  It  focuses  especially 
on  natural  language  dialog  and  speech  recognition  for  use  in  tutoring,  training,  and  performance 
aiding  systems  to  support  military  or  civilian  technicians  or  mechanics  engaged  in  inspection, 
diagnosis,  or  repair  of  aircraft,  ships,  etc..  Books,  articles,  and  standards  documents  are  briefly 
described  and  evaluated  to  provide  professionals  with  an  efficient  means  to  identify  resources  for 
use  in  lETM  development,  natural  language  dialog  prototyping,  and  evaluation  of  such  systems. 
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Preface 


This  bibliography  is  organized  into  three  major  sections:  (1)  Academic  research,  (2)  Government 
documents,  and  (3)  Standards. 

Within  the  first  section  there  are  nine  subcategories:  (a)  Hypertext,  (b)  Intelligent  Tutors, 

(c)  Performance  Aiding,  (d)  Speech  Recognition/Natural  Language  Dialog, 

(e)  Typography,  (f)  Usability  and  Evaluation,  (g)  User  Interface  Design  and  Navigation, 

(h)  Visualization,  and  (i)  Other. 

In  sections  one  and  two,  entries  are  ordered  within  each  subcategory  alphabetically  by  author’s 
last  name.  The  third  section.  Standards,  is  divided  into  two  subcategories:  (a)  Government 
[entries  219-247  are  sorted  numerically  by  Standard  number;  entries  248-252  are  sorted 
alphabetically  by  author’s  last  name],  and  (b)  ISO  [entries  253-259  are  sorted  chronologically; 
entries  260-272  are  sorted  alphabetically  by  author’s  last  name]. 

Databases  used  for  this  research  are:  Association  of  Computing  Machinery  (ACM),  FirstSearch 
(World  Catalog,  Article  First,  Dissertations,  Wilson  Select  Plus,  Readers  Guide  to  Abstracts,  and 
ECO  [collection  of  scholarly  journals]),  ProQuest  Direct,  and  Web  of  Science. 

Where  appropriate,  copies  of  papers  are  available  from:  Dr.  Jane  Siegel,  Human-Computer 
Interaction  Institute,  3603  Newell  Simon  Hall,  5000  Forbes  Avenue,  Pittsburgh,  PA  15213,  or 
via  e-mail  atjals@cs.cmu.edu. 
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1.  ACADEMIC  RESEARCH 


a.  Hypertext 

[1]  Askwall,  S.  (1985)  “Computer  Supported  Reading  Vs.  Reading  Text  on  Paper:  A  Comparison  of 

Two  Reading  Situations.”  International  Journal  of  Man-Machine  Studies  22:  425-439. 

The  effects  of  presentation  mode  on  subjects’  performance  of  a  reasoning  task  was  tested  by 
comparing  four  different  modes  of  presentation.  Subjects  were  required  to  search  and  integrate 
information  that  was  presented  in  short  texts  (22  sentences  long).  The  texts  were  presented  via  a 
VDU  (computerized  reading  situation)  or  on  a  paper  (non-computerized  reading  situation),  in 
their  entirety  or  as  separate  sentences.  Sixteen  psychology  students  participated  in  the  study. 
Reading  speed  and  accuracy  of  judgment  were  unaffected  by  presentation  medium  (VDU  or 
paper).  Moreover,  in  both  situations  search  times  were  longer  when  little  information  was 
available  and  when  search  demands  were  increased.  Negative  information  had  a  similar  effect  on 
subjects’  ratings  of  difficulty  in  the  two  situations.  The  way  information  was  searched  differed, 
however,  in  the  computerized  and  the  non-computerized  reading  situations  when  the  texts  were 
presented  as  separate  sentences.  Four  different  search  strategies  were  found;  they  were  unevenly 
distributed  in  the  two  situations.  In  the  non-computerized  situation,  subjects  searched  almost 
twice  as  much  information  as  they  did  in  the  computerized  situation.  On  the  other  hand,  in  the 
computerized  situation  search  times  were  almost  twice  as  long.  The  results  suggest  that 
psychological  studies  on  how  existing  cognitive  skills  are  applied  to  computerized  situations 
could  provide  a  valuable  source  of  information  for  designers  of  computer  systems. 

[2]  Boroni,  C.M.,  Goosey,  F.W.,  Grinder,  M.T.,  and  Ross,  J.  R.  (February  2001)  “Engaging  Students 

with  Active  Learning  Resources:  Hypertextbooks  for  the  Web.”  ACM,  SIGCSE,  pgs.  65-69. 

The  World  Wide  Web  has  mesmerized  educators  for  over  a  decade  now  with  its  tacit  promise  of 
platform  independent,  universal  educational  resource  delivery.  Indeed,  many  useful  and  exciting 
educational  tools  and  distance  learning  courseware  have  been  developed  for  the  Web.  On  the 
other  hand,  some  of  the  really  fascinating  educational  resources  -  envisioned  some  years  ago  - 
remain  elusive.  One  of  these  is  the  active  learning  hypertextbook.  In  this  paper,  the  authors 
discuss  the  concept  of  an  active  learning  hypertextbook,  point  out  the  state  of  the  art,  the 
technical  problems  to  be  surmounted,  and  the  authors  own  research  and  development  leading  to 
hypertextbook  resources  in  the  Webworks  Laboratory  at  Montana  State  University. 


[3]  Brockmann,  R.  J.  (1990)  Writing  Better  Computer  User  Documentation:  Lrom  Paper  to  Hypertext, 
Version  2.0.  New  York,  NY,  John  Wiley  &  Sons. 

Revised  edition  (first  edition,  1986)  of  a  systematic  approach  to  writing  paper  and  on-line 
documents  follows  the  process  of  creating  materials  from  the  inception  of  the  documentation 
project  to  updating  after  publication.  Designed  to  help  processing  professionals  and  technical 
writers  write  clear,  accurate  computer  user  documentation.  Presents  a  systematic  approach  to 
writing  paper  and  online  documentation.  Version  2  retains  much  essential  material  from  the  first 
edition,  while  offering  new  information  on  desktop  publishing,  CASE  tools  and  the  “software 
factory”  programming  technologies.  Also  covers  new  techniques  such  as  team  writing,  hypertext, 
and  mass  storage. 
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[4]  Chen,  C.  and  Rada,  R.  (1996)  “Interacting  With  Hypertext:  A  Meta- Analysis  of  Experimental 

Studies.”  Human-Computer  Interaction  11:  125-156. 

This  meta-analysis  compared  and  synthesized  the  results  of  23  experimental  studies  on 
hypertext.  The  analysis  was  based  on  56  pairs  of  effect  sizes  and  significance  levels  of  the 
impact  of  users,  tasks,  and  tools  on  interactions  with  hypertext.  This  analysis  focused  on  three 
factors  that  prevailingly  influence  the  use  of  hypertext:  the  cognitive  styles  and  spatial  ability  of 
users;  the  complexity  of  tasks;  and  the  structure  of  information  organization  and  the  visualization 
of  the  structure.  The  meta-analysis  found  that  this  group  of  experimental  studies  reported 
significantly  discrepant  findings,  indicating  that  substantial  differences  exist  among  individual 
experiments.  Individual  differences  in  cognition  did  not  yield  enough  evidence  to  conclude  that 
the  effect  sizes  are  significantly  apart  from  zero.  The  meta-analysis  showed  that  the  overall 
performance  of  hypertext  users  tended  to  be  more  effective  than  that  of  nonhypertext  users,  but 
the  differences  in  efficiency  measures  were  consistently  in  favor  of  nonhypertext  users.  Users 
benefited  more  from  hypertext  tools  for  open  tasks.  Overall,  the  complexity  of  tasks  has  the 
largest  combined  effect  sizes.  Graphical  maps  that  visualize  the  organization  of  hypertext  have 
significant  impact  on  the  usefulness  of  a  hypertext  system. 

[5]  Cubaud,  P.,  and  Topol,  A.  (2001).  “A  VRML-based  User  Interface  for  an  Online  Digitalized 

Antiquarian  Collection.”  ACM,  WebSD  ‘2001,  pgs.  51-59. 

This  paper  presents  a  prototypal  3D  application  for  the  access  to  a  digital  library.  It  is  mainly 
written  in  Java  and  relying  on  the  3D  engine  enclosed  in  a  VRML  browser.  By  describing  the 
basic  interactions  we  have  included  in  this  prototype  and  how  they  are  implemented,  the  authors 
show  that,  with  the  actual  Virtual  Reality  Modeling  Language  specification  (VRML97)  and  even 
with  the  new  Extented  3D  (X3D)  draft,  it  is  too  difficult  to  create  some  highly  interactive  virtual 
worlds.  The  effort  needed  to  program  basic  behaviors  with  a  script  language  raises  some 
important  problems.  Lor  solving  them,  the  authors  propose  to  work  on  adding  some  generic 
behavior  nodes  into  the  VRML  specification. 

[6]  Dietrich,  S.W.,  Suceava,  D.,  Cherukuri,  C.,  and  Urban,  S.D.  (Lebruary  2001).  “A  Reusable 

Graphical  User  Interface  for  Manipulating  Object-Oriented  Databases  using  Java  and  XML.” 
ACM.  SIGCSE  2001,  pgs.  362-366. 

This  paper  describes  the  design  and  functionality  of  a  graphical  user  interface  (GUI)  written  in 
Java  Swing  that  is  used  to  support  instructional  activities  associated  with  teaching  object- 
oriented  database  (OODB)  concepts.  The  GUI  supports  the  manipulation  of  objects  in  an  OODB, 
assuming  the  implementation  of  a  specified  interface  for  interacting  with  an  OODB.  By  using  the 
interface,  students  can  focus  on  object-oriented  design  and  programming  concepts  associated 
with  OODB  concepts  rather  than  the  development  of  a  user  interface.  Since  the  GUI  uses  the 
Extensible  Markup  Language  (XML)  for  defining  the  database  schema  and  data  import/export, 
the  use  of  the  GUI  provides  the  added  benefit  of  demonstrating  the  manner  in  which  XML 
interacts  with  database  technology. 


[7]  Konstantinou,  V.  and  Morse,  P.  (1992)  Electronic  Document  System:  Using  Automated  Hypertext 
Techniques  for  Technical  Support  Services.  Proceedings  of  the  10*  Annual  International 
Conference  on  Systems  Documentation,  SIGDOC  ‘92. 

The  Electronic  Documentation  System  was  developed  to  provide  fast,  easy,  online  information 
retrieval,  meeting  a  specific  need  for  engineering  industries.  It  addresses  the  requirements  of 
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both  the  user  and  the  author  of  electronic  documentation  and  provides  a  viewing  system  that  can 
be  used  across  a  variety  of  platforms.  It  also  takes  advantage  of  sophisticated  cross-referencing 
techniques  to  automate  the  development  of  electronic  documents. 


[8]  Landauer,  T.  Egan,  D.  Remde,  J.  Lesk,  M.  Lochbaum,  C.  and  Ketchum,  D.  Enhancing  the 

Usability  of  Text  Through  Computer  Delivery  and  Eormative  Evaluation:  the  SuperBook  Project. 
HYPERTEXT:  A  Psychological  Perspective.  C.  McKnight,  Dillon,  A.  and  Richardson,  J.  New 
York,  Ellis  Horwood:  71-136. 

The  SuperBook  project  grew  out  of  the  convergence  of  two  sources  of  interest  in  the  use  of 
computers  for  the  delivery  of  text.  The  first  was  a  set  of  research  studies  concerned  with  the 
psychology  of  information  access  by  humans,  studies  initially  done  in  the  context  of  document 
retrieval  and  interfaces  to  databases.  These  studies  attempted  to  analyze  the  factors  involved 
when  people  look  for  and  assimilate  information,  and  in  particular  to  identify  important  reasons 
for  failures.  Out  of  these  studies,  in  turn,  grew  a  number  of  schemes  for  improving  human 
performance  in  search  tasks.  These  schemes  used  computers  to  support  indexing,  querying, 
navigation  and  display  in  new  ways  that  are  difficult,  if  not  impossible,  to  accomplish  with  paper 
and  ink  technology.  The  other  source  of  the  SuperBook  concept  was  the  ongoing  evolution  of 
computer-based  methods  for  composing  and  publishing  textual  materials.  Word  processing  and 
automated  printing  technology  now  make  the  vast  majority  of  newly  produced  textual  materials 
potentially  available  in  machine-readable  form.  Moreover,  the  option  of  storing  and  distributing 
large  volumes  of  text  electronically  has  become  not  only  feasible  but  economically  attractive. 
The  authors  found  that  the  design  of  text-delivery  tools  can  depend  heavily  on  the  kind  of  text 
and  what  the  user  wants  to  do  with  it  -  no  surprise  -  and  they  illustrate  some  of  the  design  issues 
that  result.  The  authors  also  briefly  describe  ports  of  the  SuperBook  system  to  a  number  of 
different  platforms,  and  some  of  the  usability  design  problems  that  have  been  raised,  although 
not  necessarily  solved  as  yet,  by  dealing  with  hardware,  software  and  standards  constraints. 
Einally,  the  authors  describe  a  new  set  of  experiments,  including  new  prototype  delivery 
interfaces  that  incorporate  some  rather  different  techniques  from  those  expressed  in  SuperBook. 
This  last  study  is  part  of  a  cooperative  research  project  in  which  substantial  quantities  of  text  in 
the  field  of  chemistry  will  be  delivered  to  faculty  and  students  at  Cornell  University  in  electronic 
form. 


[9]  Narayanan,  N.  H.  and  Hegarty,  M.  (1998)  “On  Designing  Comprehensible  Interactive  Hypermedia 
Manuals.”  International  Journal  of  Human-Computer  Studies  48(2):  267-301. 

User’s  mental  representations  and  cognitive  strategies  can  have  a  profound  influence  on  how 
they  interact  with  computer  interfaces  (Janosky,  Smith  &  Hildreth,  1986).  However,  there  is  very 
little  research  that  elucidates  such  mental  representations  and  strategies  in  the  context  of 
interactive  hypermedia.  Eurthermore,  interface  design  for  hypermedia  information  presentation 
systems  is  rarely  driven  by  what  is  known  of  users’  mental  models  and  strategies.  This  paper 
makes  three  contributions  toward  addressing  these  problems.  Eirst,  it  describes  a  novel  cognitive 
model  of  comprehension  of  multimodal  presentations  for  the  specific  application  of  explaining 
how  machines  work,  and  proposed  guidelines  for  hypermedia  design  derived  from  this  model. 
Since  the  development  of  this  model  draws  heavily  upon  research  in  both  cognitive  science  and 
computational  modeling,  a  second  contribution  is  that  it  contains  a  detailed  review  of  literature  in 
these  fields  on  comprehension  from  static  multimodal  presentations.  Third,  it  illustrates  how 
cognitive  and  computational  modeling  is  being  used  to  inform  the  design  of  hypermedia 
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information  presentation  systems  about  machines.  This  includes  a  framework  for  empirical 
validation  of  the  model  and  evaluation  of  hypermedia  design  so  that  both  theory  and  design  can 
be  refined  iteratively. 


[10]  Nielsen,  J.  (1989)  The  Matters  that  Really  Matter  for  Hypertext  Usability.  Proceedings  of  the 
Second  Annual  ACM  Conference  on  Hypertext,  Pittsburgh,  PA,  pgs.  239-248. 

Comparison  of  92  benchmark  measurements  of  various  usability  issues  related  to  hypertext 
which  have  been  published  in  the  hypertext  literature  in  order  to  find  which  ones  have  shown  the 
largest  effects. 

[11]  Patankar,  M.  S.  (1997)  “A  Framework  for  the  Design  of  a  Voice-activated,  Intelligent,  and 
Hypermedia-based  Aircraft  Maintenance  Manual.”  Doctoral  Dissertation,  School  of  Computer 
and  Information  Sciences,  Nova  Southeastern  University. 

Federal  Aviation  Regulations  require  Aviation  Maintenance  Technicians  (AMTs)  to  refer  to 
approved  maintenance  manuals  when  performing  maintenance  on  airworthy  aircraft.  Because 
these  manuals  are  paper-based,  the  larger  the  size  of  the  aircraft,  the  more  cumbersome  are  the 
manuals.  Federal  Aviation  Administration  (FAA)  recognized  the  difficulties  associated  with  the 
use  of  large  manuals  and  conducted  studies  on  the  use  of  electronic  media  as  an  alternative  to  the 
traditional  paper  format.  However,  these  techniques  do  not  employ  any  artificial  intelligence 
technologies  and  the  user  interface  is  limited  to  either  a  keyboard  or  a  stylus  pen.  The  primary 
emphasis  of  this  research  was  to  design  a  generic  framework  that  would  allow  future 
development  of  voice-activated,  intelligent,  and  hypermedia-based  aircraft  maintenance  manuals. 
A  prototype  (VIHAMS — Voice-activated,  Intelligent,  and  Hypermedia-based  Aircraft 
Maintenance  System)  was  developed,  as  a  secondary  emphasis,  using  the  design  and 
development  techniques  that  evolved  from  this  research.  An  evolutionary  software  design 
approach  was  used  to  design  the  proposed  framework  and  the  structured  rapid  prototyping 
technique  was  used  to  produce  the  VIHAMS  prototype.  VoiceAssist  by  Creative  Labs  was  used 
to  provide  the  voice  interface  so  that  the  users  (AMTs)  could  keep  their  hands  free  to  work  on 
the  aircraft  while  maintaining  complete  control  over  the  computer  through  discrete  voice 
commands.  KnowledgePro  for  Windows,  an  expert  system  shell,  provided  “intelligence”  to  the 
prototype.  As  a  result  of  this  intelligence,  the  system  provided  expert  guidance  to  the  user.  The 
core  information  contained  in  conventional  manuals  was  available  in  a  hypermedia  format.  The 
prototype’s  operating  hardware  included  a  notebook  computer  with  a  fully  functional  audio 
system.  An  external  microphone  and  the  built-in  speaker  served  as  the  input  and  output  devices 
(along  with  the  color  monitor),  respectively.  FAA  estimates  that  U.S.  air  carriers  operated  3,991 
large  jet  aircraft  in  the  year  1996  (FAA  Aviation  Forecasts,  1987-1998).  With  an  estimate  of 
seventy  manuals  per  such  aircraft,  the  development  of  intelligent  manuals  is  expected  to  impact 
279,370  manuals  in  this  country.  Soon,  over  55  thousand  maintenance  technicians  will  be  able  to 
carry  the  seven  pound  system  to  an  aircraft,  use  voice  commands  to  access  the  aircraft’s  files  on 
the  system,  seek  assistance  from  the  expert  system  to  diagnose  the  fault,  and  obtain  instructions 
on  how  to  rectify  the  fault.  An  evolutionary  design  approach  and  the  rapid  prototyping 
techniques  were  used  to  test  the  structural  and  functional  validity  of  this  research.  Evaluators 
used  standard  testing  tools  and  evaluated  the  prototype  under  field  conditions.  They  concluded 
that  the  VIHAMS  prototype  used  a  valid  fault  diagnosis  strategy,  the  system  architecture  could 
be  used  to  develop  similar  systems  using  off-the-shelf  tools,  and  the  voice  input  system  could  be 
refined  to  improve  its  usability. 
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[12]  Weinreich,  H.,  Obendorf,  H.,  and  Lamersdorf,  W.  (August  2001).  “The  Look  of  the  Link  - 

Concepts  for  the  User  Interface  of  Extended  Hyperlinks.”  ACM,  HT  ’01,  pgs.  19-28. 

The  design  of  hypertext  systems  has  been  subject  to  intense  research.  Apparently,  one  topic  was 
mostly  neglected:  how  to  visualize  and  interact  with  link  markers.  The  paper  presents  an 
overview  of  pragmatic  historical  approaches,  and  discusses  problems  evolving  from 
sophisticated  hypertext  linking  features.  Blending  the  potential  of  an  Xlink-enhance  Web  with 
old  ideas  and  recent  GUI  techniques,  a  vision  for  browser  link  interfaces  of  the  future  is  being 
developed.  The  authors  hope  to  stimulate  the  development  of  a  standard  for  hyperlink  marker 
interfaces,  which  is  easy-to-use,  feasible  for  extended  linking  features,  and  more  consistent  with 
current  approaches. 

[13]  Zizi,  M.  and  Beaudouin-Lafon,  M.  (1994)  Accessing  Hyperdocuments  through  Interactive 
Dynamic  Maps.  Proceedings  of  the  1994  ACM  European  Conference  on  Hypermedia 
Technology. 

The  authors  proposed  a  new  navigation  paradigm  based  on  a  spatial  metaphor  to  help  users 
access  and  navigate  within  large  sets  of  documents.  This  metaphor  is  implemented  by  a 
computer  artifact  called  an  Interactive  Dynamic  Map  (IDM).  An  IDM  plays  a  role  similar  to  the 
role  of  a  real  map  with  respect  to  physical  space.  Two  types  of  IDMs  are  computed  from  the 
documents:  Topic  IDMs  represent  the  semantic  contents  of  a  set  of  documents  while  Document 
IDMs  visualize  a  subset  of  documents  such  as  those  resulting  from  a  query.  IDMs  can  be  used 
for  navigating,  browsing,  and  querying.  They  can  be  made  active,  they  can  be  customized  and 
they  can  be  shared  among  users.  The  article  presents  the  SHADOCS  document  retrieval  system 
and  describes  the  role,  use  and  generation  of  IDMs  in  SHADOCS. 


b.  Intelligent  Tutors 

[14]  Aleven,  V.A.W.M.M,  and  K.R.  Koedinger.  (2002).  “An  Effective  Metacognitive  Strategy: 
Eearning  by  Doing  and  Explaining  with  a  Computer-based  Cognitive  Tutor.”  Cognitive  Science, 

26(2). 

The  authors  investigated  whether  self-explanation  can  be  scaffolded  effectively  in  a  classroom 
environment  using  Cognitive  Tutor,  which  is  intelligent  instructional  software  that  supports 
guided  learning  by  doing.  Their  research  demonstrates  that  the  benefits  of  self-explanation  can 
be  achieved  in  a  relatively  simple  computer-based  approach  that  scales  well  for  classroom  use. 

[15]  Aleven,  V.,  and  K.R.  Koedinger.  (2000).  “The  Need  for  Tutorial  Dialog  to  Support  Self- 
Explanation.”  In  C.P.  Rose  and  R.  Ereedman  (Eds.),  Building  Dialogue  Systems  for  Tutorial 
Applications,  Papers  of  the  2000  AAAI  Pall  Symposium  (pp.  65-73).  Technical  Report  PS-00- 
01 .  Menlo  Park,  CA:  AAAI  Press. 

What  are  good  ways  of  using  natural  language  dialog  in  intelligent  tutoring  systems?  A  role  with 
high  potential  pay-off  is  to  support  the  meta-cognitive  process  of  self-explanation.  In  previous 
experiments  involving  the  PACT  Geometry  Tutor,  we  found  that  students  learn  with  greater 
understanding,  when  they  are  required  to  explain  their  solutions  steps  "by  reference",  that  is,  by 
naming  the  rule  that  was  used.  However,  the  tutor  may  be  even  more  effective  if  students  explain 
their  solution  steps  in  their  own  words  and  if  the  tutor  helps  them,  through  dialog,  to  improve 
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their  explanations.  An  exploratory  experiment  with  a  tutor  version  that  did  not  do  any  natural 
language  processing,  strongly  suggested  the  need  for  natural  language  dialog.  Without  feedback 
from  the  tutor,  students  provided  few  free-form  explanations  in  response  to  the  tutor's  prompts. 
Of  the  explanations  that  they  did  provide,  only  a  small  portion  were  correct  and  complete. 
During  the  experiment,  we  also  identified  a  number  of  dialog  strategies  that  we  plan  to 
implement  in  the  tutor. 

[16]  Anderson,  J.R.,  A.T.  Corbett,  K.R.  Koedinger,  and  R.  Pelletier.  (1995).  “Cognitive  Tutors: 

Lessons  Learned.”  The  Journal  of  the  Learning  Sciences,  4(2),  pgs.  167-207. 

This  paper  reviews  the  ten  year  history  of  tutor  development  based  on  the  ACT  theory 
(Anderson,  1983,  1993).  We  developed  production  system  models  in  ACT  of  how  students 
solved  problems  in  LISP,  geometry,  and  algebra.  Computer  tutors  were  developed  around  these 
cognitive  models.  Construction  of  these  tutors  was  guided  by  a  set  of  eight  principles  loosely 
based  on  the  ACT  theory.  Early  evaluations  of  these  tutors  usually  but  not  always  showed 
significant  achievement  gains.  Best  case  evaluations  showed  that  students  could  achieve  at  least 
the  same  level  of  proficiency  as  conventional  instruction  in  one-third  of  the  time.  Empirical 
studies  showed  that  students  were  learning  skills  in  production-rule  units  and  that  the  best 
tutorial  interaction  style  was  one  in  which  the  tutor  provides  immediate  feedback,  consisting  of 
short  and  directed  error  messages.  The  tutors  appear  to  work  better  if  they  present  themselves  to 
students  as  non  human  tools  to  assist  learning  rather  than  as  emulations  of  human  tutors. 
Students  working  with  these  tutors  display  transfer  to  other  environments  to  the  degree  that  they 
can  map  the  tutor  environment  into  the  test  environment.  These  experiences  have  coalesced  into 
a  new  system  for  developing  and  deploying  tutors.  This  system  involves  first  selecting  a 
problem-solving  interface,  then  constructing  a  curriculum  under  the  guidance  of  a  domain  expert, 
then  designing  a  cognitive  model  for  solving  problems  in  that  environment,  then  building 
instruction  around  the  productions  in  that  model,  and  finally  deploying  the  tutor  in  the  classroom. 
New  tutors  are  being  built  in  this  system  to  achieve  the  NCTM  standards  for  high  school 
mathematics  in  an  urban  setting. 

[17]  Baker,  R.S.,  A.T.  Corbett,  and  K.R.  Koedinger.  (2002).  “The  Resilience  of  Overgeneralization  of 
Knowledge  about  Data  Representations.”  Presented  at  the  American  Educational  Research 
Association  Conference . 

In  this  paper,  the  authors  present  a  study  that  they  conducted  to  explore  the  depth  of  transfer  and 
to  investigate  methods  for  reducing  its  occurrence  in  generation,  where  it  is  clearly  inappropriate. 
Given  the  strength  of  the  student  desire  to  choose  axes  appropriate  to  a  bar  graph  when 
generating  scatterplots  and  histograms,  the  authors  decided  to  investigate  whether  there  were 
ways  to  induce  these  students  to  instead  draw  the  correct  axes,  and  if  there  was  further 
mistransfer  of  bar  graph  knowledge  once  the  students  had  the  correct  exes,  as  the  effect  in  Eehrer 
et  al  suggests.  Therefore,  the  authors  chose  the  general  intervention  of  drawing  the  students’ 
attention  to  the  variables,  in  order  to  attempt  to  express  each  of  these  factors. 

[18]  Baker,  R.S.,  A.T.  Corbett,  and  K.R.  Koedinger.  “Toward  a  Model  of  Eearning  Data 
Representations.” 

The  use  of  graphs  to  represent  and  reason  about  data  is  of  growing  importance  in  pre-high  school 
mathematics  curricula.  This  study  examines  middle  school  students’  skills  in  reasoning  about 
three  graphical  representations:  histograms,  scatterplots  and  stem-and-leaf  plots.  Students  were 
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asked  to  interpret  graphs,  select  an  appropriate  graph  type  to  represent  a  relationship  and  to 
generate  graphs.  Accuracy  levels  varied  substantially  across  the  three  tasks  and  three  graph 
types.  The  overall  pattern  of  results  is  largely  explained  by  the  varying  ease  of  transfer  of  student 
knowledge  from  a  simpler  graph  type,  based  on  surface  similarity. 

[19]  Koedinger,  K.R.,  J.R.  Anderson,  W.H.  Hadley,  and  M.  Mark.  (1997).  “Intelligent  Tutoring  Goes 
to  School  in  the  Big  City.”  International  Journal  of  Artificial  Intelligence  in  Education,  8,  30-43. 
This  paper  reports  on  a  large-scale  experiment  introducing  and  evaluating  intelligent  tutoring  in 
an  urban  High  School  setting.  Critical  to  the  success  of  this  project  has  been  a  client-centered 
design  approach  that  has  matched  their  client's  expertise  in  curricular  objectives  and  classroom 
teaching  with  their  expertise  in  artificial  intelligence  and  cognitive  psychology.  The  Pittsburgh 
Urban  Mathematics  Project  (PUMP)  has  produced  an  algebra  curriculum  that  is  centrally  focused 
on  mathematical  analysis  of  real  world  situations  and  the  use  of  computational  tools.  The  authors 
have  built  an  intelligent  tutor,  called  PAT,  that  supports  this  curriculum  and  has  been  made  a 
regular  part  of  9th  grade  Algebra  in  3  Pittsburgh  schools.  In  the  1993-94  school  year,  the  authors 
evaluated  the  effect  of  the  PUMP  curriculum  and  PAT  tutor  use.  On  average  the  470  students  in 
experimental  classes  outperformed  students  in  comparison  classes  by  15%  on  standardized  tests 
and  100%  on  tests  targeting  the  PUMP  objectives.  This  study  provides  further  evidence  that 
laboratory  tutoring  systems  can  be  scaled  up  and  made  to  work,  both  technically  and 
pedagogically,  in  real  and  unforgiving  settings  like  urban  high  schools. 

[20]  Koedinger,  K.R.,  and  M.J.  Nathan,  (in  press).  “The  Real  Story  Behind  Story  Problems:  Effects  of 
Representations  on  Quantitative  Reasoning.”  International  Journal  of  the  Learning  Sciences. 

The  authors  explored  how  differences  in  problem  representations  change  both  the  performance 
and  underlying  cognitive  processes  of  beginning  algebra  students  engaged  in  quantitative 
reasoning.  Contrary  to  beliefs  held  by  practitioners  and  researchers  in  mathematics  education,  the 
authors  found  that  students  were  more  successful  solving  simple  algebra  story  problems  than 
solving  mathematically  equivalent  equations.  Contrary  to  some  views  of  situated  cognition,  this 
result  is  not  simply  a  consequence  of  situated  world  knowledge  facilitating  problem  solving 
performance,  but  rather  a  consequence  of  student  difficulties  with  comprehending  the  formal 
symbolic  representation  of  quantitative  relations.  The  authors  draw  on  analyses  of  students' 
strategies  and  errors  as  the  basis  for  a  cognitive  process  explanation  of  when,  why,  and  how 
differences  in  problem  representation  affect  problem  solving. 


c.  Performance  Aiding 

[21]  Brusilovsky,  P.,  and  Cooper,  D.W.  (January  13,  2002).  “Domain,  Task,  and  User  Models  for  an 
Adaptive  Hypermedia  Performance  Support  System.”  ACM,  lUI  ’02,  pgs.  23-30. 

Electronic  Performance  Support  Systems  (EPSS)  is  a  challenging  application  area  for  developing 
intelligent  interfaces.  Some  possible  scenarios  for  using  domain,  task,  and  user  models  for 
adaptive  performance  support  were  explored  in  the  context  of  the  Adaptive  Diagnostics  and 
Personalized  Technical  Support  (ADAPTS)  project.  ADAPTS  provides  an  intelligent,  adaptive 
EPSS  for  maintaining  complex  equipment. 
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[22]  Cavazza,  M.,  F.  Charles,  and  SJ.  Mead.  (July  15-19,  2002).  “Interacting  with  Virtual  Characters 

in  Interactive  Storytelling.”  AAMA5  ’02,  ACM:  318-325. 

In  recent  years,  several  paradigms  have  emerged  for  interactive  storytelling.  In  character-based 
storytelling,  plot  generation  is  based  on  the  behavior  of  autonomous  characters.  In  this  paper,  the 
authors  describe  user  interaction  in  a  fully  implemented  prototype  of  an  interactive  storytelling 
system.  The  authors  describe  the  planning  techniques  used  to  control  autonomous  characters, 
which  derive  from  HTN  planning.  The  hierarchical  task  network  representing  a  characters’ 
potential  behavior  constitute  a  target  for  user  intervention,  both  in  terms  of  narrative  goals  and  in 
terms  of  physical  actions  carried  out  on  stage.  The  authors  introduce  two  different  mechanisms 
for  user  interaction:  direct  physical  interaction  with  virtual  objects  and  interaction  with  synthetic 
characters  through  speech  understanding.  The  authors  illustrate  these  functionalities  with 
examples  of  system-generated  behavior  and  conclude  with  a  discussion  of  scalability  issues. 

[23]  Franklin,  D.,  Budzik,  J.,  and  Hammond,  K.  (January  13,  2002).  “Plan-based  Interfaces:  Keeping 
Track  of  User  Tasks  and  Acting  to  Cooperate.”  ACM,  lUI  ’02,  pgs.  79-86. 

The  ability  to  reason  about  the  activity  of  a  user  is  crucial  to  the  implementation  of  any 
Intelligent  User  Interface.  It  if  is  able  to  recognize  what  a  user  is  doing,  a  computer  can  act  to 
cooperate.  Most  computer  systems  limit  themselves  to  command-response  interactions  -  their 
trivial  understandings  of  their  users  cannot  support  a  more  complicated  interaction.  However,  by 
looking  at  the  tasks  that  their  users  are  performing  and  reasoning  about  sequences  of  actions,  a 
computer  system  can  provide  a  more  interesting  level  of  interaction  that  is  more  efficient  and 
does  not  demand  as  much  of  its  users.  Furthermore,  the  understanding  of  the  user’s  activity 
provides  a  context  within  which  to  better  understand  future  actions  and  to  tune  the  sensing 
systems  to  look  and  listen  for  the  actions  that  the  user  is  most  likely  to  take  next.  Finally,  in 
many  domains,  such  computer  systems  can  recognize  user  tasks  and  act  to  cooperate  without 
requiring  a  deep,  goal-oriented  understanding.  In  this  paper,  the  authors  look  at  the  process-based 
interface  used  in  the  Intelligent  Classroom,  focusing  on  how  a  human  lecturer  can  control  it  by 
simply  going  about  her  presentation.  Also,  the  authors  look  at  how  the  general  ideas  have  been 
adapted  to  Jabberwocky,  a  speech-based  interface  to  Microsoft  PowerPoint  that  automatically 
switches  slides,  and  how  they  are  being  applied  to  extend  the  functionality  of  Watson,  an 
autonomous  web  research  tool  that  uses  the  document  a  user  is  viewing  as  a  search  context. 

[24]  Hombaek,  K.,  and  Frokjaer,  E.  (March  2001).  “Reading  of  Electronic  Documents:  The  Usability 
of  Einear,  Eisheye,  and  Overview-i-Detail  Interfaces.”  ACM,  SIGCHI  ’01,  3:  293-300. 

Reading  of  electronic  documents  is  becoming  increasingly  important  as  more  information  is 
disseminated  electronically.  The  authors  present  an  experiment  that  compares  the  usability  of  a 
linear,  a  fisheye,  and  an  overview-i-detail  interface  for  electronic  documents.  Essays  written  using 
the  overview-i-detail  interface  received  higher  grades,  while  subjects  using  the  fisheye  interface 
read  documents  faster.  However,  subjects  used  more  time  to  answers  questions  with  the 
overview-i-detail  interface.  The  most  common  interface  in  practical  use,  the  linear  interface,  is 
found  to  be  inferior  to  the  fisheye  and  overview-i-detail  interfaces  regarding  most  aspects  of 
usability.  The  authors  recommend  using  overview-i-detail  interfaces  for  electronic  documents, 
while  fisheye  interfaces  mainly  should  be  considered  for  time-critical  tasks. 
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[25]  Osborne,  D.  J.  and  Holton,  D.  (1988).  “Reading  From  Screen  Versus  Paper:  There  Is  No 
Difference.”  International  Journal  of  Man-Machine  Studies  28:  1-9. 

This  paper  considers  the  effect  of  presentation  medium  on  reading  speed  and  comprehension.  By 
directly  comparing  performance  using  screen  and  paper  presentations,  it  examines  the  argument 
that  it  takes  longer  to  read  from  a  screen-based  display  than  from  paper,  and  that  comprehension 
will  be  lower.  The  hypothesis  is  also  tested  that  it  takes  longer  to  read  light  characters  on  a  dark 
background  compared  with  dark  characters  on  a  light  background,  and  that  comprehension  will 
be  lower  with  light-character  displays.  Altogether  four  conditions  were  used,  with  two  passages 
read  in  each  condition:  screen  with  dark  characters,  screen  with  light  characters,  paper  with  dark 
characters,  and  paper  with  light  characters.  Subjects  also  ranked  the  four  conditions  for 
preference.  No  significant  difference  was  found  in  either  reading  speed  or  comprehension 
between  screen  and  paper,  or  between  dark  and  light  character  displays.  Some  preference 
differences  were  found,  however.  Reasons  for  the  lack  of  reading  and  comprehension  differences 
are  discussed,  and  it  is  argued  that  this  reflects  the  close  attention  to  experimental  detail  paid  in 
the  present  experiment,  which  has  often  been  missing  in  past  studies. 


d.  Speech  Recognition  /  Natural  Language  Dialog 

[26]  Allan,  J.,  and  H.  Raghavan.  (2002).  “Using  Part-of-speech  Patterns  to  Reduce  Query  Ambiguity.” 
SIGIR  ’02,  ACM:  307-314. 

Query  ambiguity  is  a  generally  recognized  problem,  particularly  in  Web  environments  where 
queries  are  commonly  only  one  or  two  words  in  length.  In  this  study,  the  authors  explore  one 
technique  that  finds  commonly  occurring  patterns  of  parts  of  speech  near  a  one- word  query  and 
allows  them  to  be  transformed  into  clarification  questions.  The  authors  use  a  technique  derived 
from  statistical  language  modeling  to  show  that  the  clarification  queries  will  reduce  ambiguity 
much  of  the  time,  and  often  quite  substantially. 

[27]  Boyce,  S.J.,  and  A.L.  Gorin.  “User  Interface  Issues  for  Natural  Spoken  Dialog  Systems.”  AT&T 
Laboratories. 

The  authors  are  interested  in  building  machines  which  can  understand  and  act  upon  fluently 
spoken  language.  This  is  often  not  a  single  interaction,  but  rather  involves  a  dialog  to  negotiate 
the  proper  outcome.  The  authors  goal  was  to  make  such  human-computer  dialogs  as  natural  as 
possible,  so  as  to  enable  large  populations  of  non-experts  to  use  such  systems.  There  are  many 
user  interface  issues  which  then  arise.  The  authors  focus  on  a  particular  experimental  vehicle, 
that  of  automatically  routing  telephone  calls  based  on  a  user’s  fluently  spoken  request  to  “How 
may  1  help  you?” .  The  authors  describe  several  elements  that  are  necessary  in  human-computer 
dialog  which  do  not  have  ready  analogs  in  the  human-human  case.  Finally,  the  authors  present 
preliminary  experimental  results  evaluating  user  interface  options  for  these  elements. 

[28]  Dobroth,  K.  (February  /  March  2000).  “Beyond  Natural:  Adding  Appeal  to  Speech  Recognition 
Applications?”  SpeechTechMag.com 
http://www.speechtechmag.com/pub/5_2/cover/191-l.html 

Although  humans  engage  in  conversation  with  each  other  effortlessly,  conversation  is  incredibly 
complex.  For  decades  now,  researchers  have  been  trying  to  develop  algorithms  that  capture  the 
essence  of  conversation,  and  yet  there  still  does  not  seem  to  be  any  commercially  available 
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speech  recognition  systems  that  really  behave  as  a  person  would  in  a  conversation.  If  users 
expect  that  talking  to  a  speech  recognition  application  will  be  just  like  talking  to  a  person,  it  is 
not  clear  that  applications  could  ever  measure  up.  But,  for  speech  applications  to  be  good, 
successful  applications,  do  they  have  to  resemble  human  conversation?  And,  should  they? 

[29]  Ezzat,  T.,  G.  Geiger,  and  T.  Poggio.  (2002).  “Trainable  Videorealistic  Speech  Animation.”  ACM: 

388-398. 

The  authors  describe  how  to  create  with  machine  learning  techniques  a  generative, 
Videorealistic,  speech  animation  module.  A  human  subject  is  first  recorded  using  a  videocamera 
as  he/she  utters  a  predetermined  speech  corpus.  After  processing  the  corpus  automatically,  a 
visual  speech  module  is  learned  form  the  data  that  is  capable  of  synthesizing  the  human  subject’s 
mouth  uttering  entirely  novel  utterances  that  were  not  recorded  in  the  original  video.  The 
synthesized  utterance  is  re-composited  onto  a  background  sequence  which  contains  natural  head 
and  eye  movement.  The  final  output  is  videorealistic  in  the  sense  that  it  looks  like  a  video  camera 
recording  of  the  subject.  At  run  time,  the  input  to  the  system  can  be  either  real  audio  sequences 
or  synthetic  audio  produced  by  a  text-to-speech  system,  as  long  as  they  have  been  phonetically 
aligned.  The  two  key  contributions  of  this  paper  are  (1)  a  variant  of  the  multidimensional 
morphable  model  (MMM)  to  synthesize  new,  previously  unseen  mouth  configurations  from  a 
small  set  of  mouth  image  prototypes;  and  (2)  a  trajectory  synthesis  technique  based  on 
regularization,  which  is  automatically  trained  from  the  recorded  video  corpus,  and  which  is 
capable  of  synthesizing  trajectories  in  MMM  space  corresponding  to  any  desired  utterance. 

[30]  Hakulinen,  J.,  and  M.  Turunen.  (1999).  “Prosodic  Features  for  Speech  User  Interfaces.”  ACHCI 

’99:  1-7. 

The  designing  of  system  utterances  is  a  very  crucial  part  of  speech  user  interfaces,  especially  if 
speech  synthesis  is  used.  Although  speech  synthesis  is  quite  intelligible  in  well  formed  and 
simple  sentences  it  is  very  difficult  for  users  to  understand  when  complex  structural  elements 
like  tables  are  spoken.  Furthermore,  most  users  do  not  like  the  way  synthesizers  use  prosody. 
Most  of  the  previous  research  has  focused  on  what  information  should  be  presented  to  the  user. 
Recent  research  has  also  brought  up  the  question  of  how  this  information  should  be  presented.  In 
order  to  improve  intelligibility  and  naturalness  of  synthetic  speech  the  authors  arranged  an 
experiment  to  find  new  ways  to  use  prosody.  In  this  experiment  subjects  listened  to  three  human 
readers  and  a  speech  synthesizer  reading  system  utterances  from  their  e-mail  system.  Questions 
were  asked  to  measure  how  well  the  utterances  were  understood.  Subjective  evaluations  of  the 
voices  were  also  collected.  The  authors  used  the  results  to  find  those  prosodic  elements  that  help 
users  to  better  understand  what  they  are  hearing.  Pauses  were  found  to  make  a  significant 
difference  in  comprehension.  Good  variation  in  pitch  and  speech  seem  to  make  a  voice  more 
pleasant  to  listen  to  but  have  only  minor  positive  effect  on  comprehension. 

[31]  Hakulinen,  J.,  M.  Turunen,  and  K.-J.  Raiha.  “The  Use  of  Prosodic  Features  to  Help  Users  Extract 
Information  From  Structured  Elements  in  Spoken  Dialogue  Systems.”  Human-Computer 
Interaction  Group,  Department  of  Computer  Science,  University  of  Tampere. 

Most  of  the  previous  research  on  speech  user  interfaces  has  focused  on  what  information  should 
be  presented  to  the  user.  Equally  important  is  the  question  of  how  this  information  should  be 
presented.  Although  speech  synthesis  is  quite  intelligible  in  well  formed  and  simple  sentences,  it 
may  be  very  difficult  to  understand  when  complex  structural  elements,  like  tables  or  URFs,  are 
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spoken.  The  authors  arranged  a  controlled  experiment  to  identify  the  prosodic  features  that  affect 
the  intelligibility  and  pleasantness  of  synthetic  speech.  Pauses  were  found  to  make  a  significant 
difference  in  comprehension.  Good  variation  in  pitch  and  rate  seem  to  make  a  voice  more 
pleasant  to  listen  to  but  have  only  minor  positive  effect  on  comprehension.  The  authors  analyzed 
the  exact  ways  in  which  human  readers  used  prosodic  elements  so  that  they  could  construct 
unique  and  human  like  computer  ‘persons’  for  spoken  dialogue  applications. 

[32]  Huber,  A.,  and  B.  Ludwig.  (July  15-19,  2002).  “Users  Talk  to  their  Model  Trains:  Interaction  with 

a  Speech-based  Multi- Agent  System.”  AAMAS  ’02,  ACM:  800-801. 

This  paper  presents  an  interactive  multi-agent  system  (MAS)  for  controlling  a  model  train 
installation.  User  intentions  can  be  communicated  to  the  system  via  speech  and  are  processed  by 
hierarchically  organized  autonomous  agents  with  distributed  knowledge  and  responsibilities  in 
the  application  domain.  Agents  on  deeper  levels  in  the  hierarchy  execute,  observe,  and  refine 
plans  constructed  by  higher- level  agents.  Local  planning  and  negotiating  with  the  user  resolve 
conflicts  between  requirements  of  tasks  and  the  current  application  state.  For  this  purpose  a 
multi-model  natural  language  dialogue  system  is  incorporated  as  the  user  interface. 

[33]  lovine,  J.  (May  2002).  “Speech-Controlled  Robotic  Arm.”  Poptronics,  pgs.  17-20. 

In  April  2002,  the  author  built  a  PC  interface  for  the  OWI  robotic  arm.  Now  the  author  had 
added  a  speech-recognition  interface.  The  interface  uses  the  speech-recognition  kit.  This 
interface  fits  between  the  speech-recognition  circuit  and  the  robotic  arm.  It  allows  one  to  control 
all  the  robotic  arm  functions  using  verbal  commands.  The  robotic  arm  has  ten  wired  movement 
functions.  Required  is  one  verbal  command  for  each  function,  plus  one  additional  command  for 
stopping  all  functions  (movement).  In  total,  there  are  11  verbal  commands.  The  speech- 
recognition  kit  can  recognize  up  to  40  words.  The  second  revision  of  the  kit  allows  the  user  to 
choose,  via  a  jumper  setting,  between  recognizing  40  one-second  words,  or  20  two-second 
words;  that  setting  improves  the  word  recognition  accuracy. 

[34]  Johnson,  M.T.  (August  2000).  “Incorporating  Prosodic  Information  and  Language  Structure  into 
Speech  Recognition  Systems.”  Dissertation,  Purdue  University. 

Some  of  the  major  research  issues  in  the  field  of  speech  recognition  revolve  around  methods  of 
incorporating  additional  knowledge  sources,  beyond  the  short-time  spectral  information  of  the 
speech  signal,  into  the  recognition  process.  These  knowledge  sources,  which  may  include 
information  about  prosody,  language  structures,  semantics,  and  dialogue  context,  are  difficult  to 
quantify  with  regard  to  the  task  of  language  understanding,  and  are  even  more  difficult  to 
interface  with  the  statistically  motivated  architectures  such  as  Hidden  Markov  Models  that  are 
used  for  acoustic  processing.  The  fundamental  goal  of  this  research  is  to  further  our 
understanding  of  how  to  incorporate  prosodic  features  and  language  structure  into  recognition 
systems.  Each  of  these  two  domains  has  proven  to  be  particularly  difficult  to  use  effectively, 
especially  for  speaker  independent  tasks  where  most  of  the  elements  of  these  knowledge 
domains  vary  significantly  between  speakers.  To  accomplish  this  goal,  the  author  looks  at  the 
effectiveness  of  using  word  graphs  as  an  interface  mechanism  between  recognition  and  language 
systems,  develop  the  Observation  Dependent  Hidden  Markov  Model  (ODHMM)  which  is  able  to 
adaptively  alter  transition  probabilities  based  on  the  dynamics  of  spectral  features,  and  apply 
temporal  and  suprasegmental  information  to  the  task  of  segmenting  audio  classes  for  broadcast 
news  transcription. 
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[35]  Jurafsky,  D.,  and  J.H.  Martin.  (2000).  Speech  and  Language  Processing:  An  Introduction  to 
Natural  Language  Processing,  Speech  Recognition,  and  Computational  Linguistics.  Prentice- 
Hall. 

Part  I,  “Words”,  introduces  concepts  related  to  the  processing  of  words:  phonetics,  phonology, 
morphology,  and  algorithms  used  to  process  them:  finite  automata,  finite  transducers,  weighted 
transducers,  N-grams,  and  Hidden  Markov  Models.  Part  II,  “Syntax”,  introduces  parts-of-speech 
and  phrase  structure  grammars  for  English  and  gives  essential  algorithms  for  processing  word 
classes  and  structured  relationships  among  words:  part-of-speech  taggers  based  on  HMMs  and 
transformation-based  learning,  the  CYK  and  Earley  algorithms  for  parsing,  unification  and  typed 
feature  structures,  lexicalized  and  probabilistic  parsing,  and  analytical  tools  like  the  Chomsky 
hierarchy  and  pumping  lemma.  Part  III,  “Semantics”,  introduces  first  order  predicate  calculus 
and  other  ways  of  representing  meaning,  several  approaches  to  compositional  semantic  analysis, 
along  with  applications  to  information  retrieval,  information  extraction,  speech  understanding, 
and  machine  translation.  Part  IV,  “Pragmatics”,  covers  reference  resolution  and  discourse 
structure  and  coherence,  spoken  dialogue  phenomena  like  dialogue  and  speech  act  modeling, 
dialogue  structure  and  coherence,  and  dialogue  managers,  as  well  as  a  comprehensive  treatment 
of  natural  language  generation  and  of  machine  translation. 

[36]  Kaur,  M.  (October  2000).  “Integration  of  Gaze  and  Speech  for  Multimodal  Human  Computer 
Interaction.”  Dissertation,  New  Brunswick,  Rutgers,  The  State  University  of  New  Jersey. 

Most  commonly  used  human-computer  interfaces  do  not  take  advantage  of  the  many 
communication  channels  humans  use  to  communicate  in  verbal  and  non-verbal  ways.  The 
Rutgers  University  CAIP  Center,  under  the  NSE  STIMUEATE  program,  has  been  conducting 
research  to  establish,  quantify,  and  evaluate  techniques  for  designing  synergistic  combinations  of 
human-machine  communication  modalities  like  sight,  sound,  and  touch.  An  initial  system  using 
these  modalities  has  been  implemented  at  CAIP,  and  it  has  been  seen  that  even  with  our 
simplistic  integration  scheme  and  imperfect  component  technologies,  there  are  obvious 
performance  advantages  to  be  gained  from  the  use  of  multiple  modalities.  The  research  described 
in  this  thesis  is  a  systematic  evaluation  and  characterization  of  gaze  as  an  input  modality,  and  its 
integration  with  speech.  The  overall  goal  of  the  research  was  to  explore  the  use  of  gaze  and 
speech  as  input  modalities  for  HCI,  and  to  understand  the  natural  integration  patterns  typically 
occurring  in  the  combined  use  of  the  two.  Exhaustive  characterization  of  the  use  of  gaze  as  an 
input  modality  for  human-computer  interaction  has  been  done.  Relationship  between  object 
selection  times  with  distance,  size,  and  index  of  difficulty,  as  defined  by  Eitts  law,  have  been 
studied.  The  speech  and  gaze  experiments  described  in  this  work  provide  detailed  timing 
correlations  of  speech  and  gaze,  both  in  a  natural  environment,  and  a  computer-based  system 
which  can  be  used  to  answer  the  questions  of  when  and  how  to  integrate  the  two  modalities.  The 
integration  of  speech  and  gaze  has  been  studied  under  linguistically  different  command 
structures  like  the  use  of  labels  (Move  A  to  i),  descriptors  (Move  triangular  bar  to  triangular 
slot)  and  anaphora  (Move  This  There).  It  has  been  found  that  there  are  fundamental  relationships 
between  gaze  and  speech  events,  e.g.,  gaze  always  precedes  speech,  though  the  time  is  seen  to 
vary  with  command  structure.  The  effect  of  command  structure  is  seen  to  decrease  in  a 
computer-based  environment.  Also,  it  has  been  demonstrated  that  the  detailed  knowledge  of  the 
timing  relationship  of  the  speech  and  gaze  patterns  for  command  specification  and  command 
execution  can  be  used  for  error  resolution  for  a  more  robust  multimodal  interface. 
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[37]  Lin,  W.-H.,  and  A.G.  Hauptmann.  (July  13-17,  2002).  “A  Wearable  Digital  Library  of  Personal 
Conversations.”  JCDL  ’02,  ACM:  277-278. 

The  authors  have  developed  a  wearable,  personalized  digital  library  system,  which  unobtrusively 
records  the  wearer’s  part  of  a  conversation,  recognizes  the  face  of  the  current  dialog  partner  and 
remembers  his/her  voice.  The  next  time  the  system  sees  the  same  person  and  hears  the  same 
voice;  it  can  replay  parts  of  the  last  conversation  in  compressed  form.  Results  from  a  prototype 
system  show  the  effectiveness  of  combining  of  face  recognition  and  speaker  identification  for 
retrieving  conversations. 

[38]  ManE,  A.,  S.  Boyce,  D.  Karis,  and  N.  Yankelovich.  (1996).  “Designing  the  User  Interface  for 

Speech  Recognition  Applications:  A  CHI  96  Workshop.”  Presented  at  a  workshop  of  the  ACM 
SICCHI  Conference. 

http://www.acm.Org/sigchi/bulletin/I996.4/boyce.html 

The  progress  made  over  the  last  few  years  in  speech  technology  has  generated  new  opportunities 
and  new  challenges  for  designers.  One  common  need  that  was  expressed  by  the  participants  is 
for  a  rich  set  of  tools  for  specifying  and  prototyping  user  interfaces,  and  for  a  rich  exchange  of 
information  between  the  part  of  the  application  that  governs  the  user  interface  and  the  part  that 
analyzes  the  user's  speech.  The  workshop  successfully  covered  a  gamut  of  issues  that  underlie 
the  design  of  speech-based  interfaces.  One  common  sentiment  was  that  after  having  this 
theoretical  discussion  as  a  background,  the  authors  can  now  focus  more  on  the  practical 
challenges  of  designing  speech  based  interfaces. 

[39]  McTear,  M.F.  (2002).  “Spoken  Dialogue  Technology:  Enabling  the  Conversational  User 
Interface.”  ACM  Computing  Surveys  34:  90-169. 

Spoken  dialogue  systems  allow  users  to  interact  with  computer-based  applications  such  as 
databases  and  expert  systems  by  using  natural  spoken  language.  The  origins  of  spoken  dialogue 
systems  can  be  traced  back  to  Artificial  Intelligence  research  in  the  1950s  concerned  with 
developing  conversational  interfaces.  However,  it  is  only  within  the  last  decade  or  so,  with  major 
advances  in  speech  technology,  that  large-scale  working  systems  have  been  developed  and,  in 
some  cases,  introduced  into  commercial  environments.  As  a  result  many  telecommunications  and 
software  companies  have  become  aware  of  the  potential  for  spoken  dialogue  technology  to 
provide  solutions  in  newly  developing  areas  such  as  computer-telephony  integration.  Voice 
portals,  which  provide  a  speech-based  interface  between  a  telephone  user  and  Web-based 
services,  are  the  most  recent  application  of  spoken  dialogue  technology.  This  article  describes 
the  main  components  of  the  technology  -  speech  recognition,  language  understanding,  dialogue 
management,  communication  with  an  external  source  such  as  a  database,  language  generation, 
speech  synthesis  -  and  shows  how  these  component  technologies  can  be  integrated  into  a  spoken 
dialogue  system.  This  article  describes  in  detail  the  methods  that  have  been  adopted  in  some 
well-known  dialogue  systems,  explores  different  system  architectures,  considers  issues  of 
specification,  design,  and  evaluation,  reviews  some  currently  available  dialogue  development 
toolkits,  and  outlines  prospects  for  future  development. 
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[40]  Nichols,  J.,  B.  Myers,  T.K.  Harris,  R.  Rosenfeld,  S.  Shriver,  M.  Higgins,  and  J.  Hughes.  (October 

14-16,  2002).  “Requirements  for  Automatically  Generating  Multi-Modal  Interfaces  for  Complex 
Appliances.”  Fourth  IEEE  ICMI  ’02  -  International  Conference  on  Multimodal  Interfaces, 
Pittsburgh,  PA,  pgs.  377-382. 

Several  industrial  and  academic  research  groups  are  working  to  simplify  the  control  of 
appliances  and  services  by  creating  a  truly  universal  remote  control.  Unlike  the  preprogrammed 
remote  controls  available  today,  these  new  controllers  download  a  specification  from  the 
appliance  or  service  and  use  it  to  automatically  generate  a  remote  control  interface.  This 
promises  to  be  a  useful  approach  because  the  specification  can  be  made  detailed  enough  to 
generate  both  speech  and  graphical  interfaces.  Unfortunately,  generating  good  user  interfaces  can 
be  difficult.  Based  on  user  studies  and  prototype  implementations,  this  paper  presents  a  set  of 
requirements  that  the  authors  have  found  are  needed  for  automatic  interface  generation  systems 
to  create  high-quality  user  interfaces. 

[41]  Oard,  D.W.  (June/July  2000).  “User  Interface  Design  for  Speech-Based  Retrieval.”  Bulletin  of  the 
American  Society  for  Information  Science,  pgs.  20-22. 

It  is  now  quite  practical  to  apply  speech-based  retrieval  techniques  to  collections  containing 
several  thousand  hours  of  retrieved  audio,  and  Moore’s  Law  assures  us  that  those  numbers  will 
grow  rapidly  over  the  next  few  years.  If  we  are  to  make  the  best  use  of  these  emerging 
capabilities,  we  will  need  to  devote  increased  attention  to  the  design  of  user  interfaces  that 
support  effective  search  strategies.  This  will  undoubtedly  be  an  interactive  process,  since  we 
cannot  hope  to  understand  which  search  strategies  will  be  most  effective  until  we  have  a  rich  set 
of  user  interfaces  with  which  to  experiment.  This  article  briefly  reviews  the  process  of  searching 
electronic  text  collections  and  then  describes  how  the  process  can  be  adapted  to  support 
searching  large  audio  collections  based  on  speech  contained  in  those  collections. 

[42]  Oh,  A.H.,  and  A. I.  Rudnicky.  “Stochastic  Natural  Language  Generation  for  Spoken  Dialog 
Systems.”  Article  submitted  to  Computer  Speech  and  Language._ 

The  authors  describe  a  corpus-based  approach  to  natural  language  generation  (NLG).  The 
approach  has  been  implemented  as  a  component  of  a  spoken  dialog  system  and  a  series  of 
evaluations  were  carried  out.  The  system  uses  n-gram  language  models,  which  have  been  found 
useful  in  other  language  technology  applications,  in  a  generative  mode.  It  is  not  yet  clear  whether 
the  simple  n-grams  can  adequately  model  human  language  generation  in  general,  but  we  show 
that  we  can  successfully  apply  this  ubiquitous  modeling  technique  to  the  task  of  natural  language 
generation  for  spoken  dialog  systems.  In  this  paper,  the  authors  discuss  applying  corpus-based 
stochastic  language  generation  at  two  levels:  content  selection  and  sentence  planning/realization. 
At  the  content  selection  level,  output  utterances  are  modeled  by  bigrams,  and  the  appropriate 
attributes  are  chosen  using  bigram  statistics.  In  sentence  planning  and  realization,  corpus 
utterances  are  modeled  by  n-grams  of  varying  length,  and  new  utterances  are  generated 
stochastically.  Through  this  work,  the  authors  show  that  a  simple  statistical  model  alone  can 
generate  appropriate  language  for  a  spoken  dialog  system.  The  results  describe  a  promising 
avenue  for  using  a  statistical  approach  in  future  NLG  systems. 
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[43]  Oviatt,  S.,  and  P.  Cohen.  (2000).  “Designing  the  User  Interface  for  Multimodal  Speech  and  Pen- 
Based  Gesture  Applications:  State-of-the-Art  Systems  and  Future  Research  Directions.”  Human- 
Computer  Interaction  15,  pgs.  263-322. 

The  growing  interest  in  multimodal  interface  design  is  inspired  in  large  part  by  the  goals  of 
supporting  more  transparent,  flexible,  efficient,  and  powerfully  expressive  means  of  human- 
computer  interaction  than  in  the  past.  Multimodal  interfaces  are  expected  to  support  a  wider 
range  of  diverse  populations,  be  usable  by  a  broader  spectrum  of  the  average  population,  and 
function  more  reliably  under  realistic  and  challenging  usage  conditions.  In  this  article,  the 
authors  summarize  the  emerging  architectural  approaches  for  interpreting  speech  and  pen-based 
gestural  input  in  a  robust  manner  -  including  early  and  lat  fusion  approaches,  and  the  new  hybrid 
symbolic-statistical  approach.  The  authors  also  describe  a  diverse  collection  of  state-of-the-art 
multimodal  systems  that  process  users’  spoken  and  gestural  input.  These  applications  range  from 
map-based  and  virtual  reality  systems  for  engaging  in  simulations  and  training,  to  field  medec 
systems  for  mobile  use  in  noisy  environments,  to  web-based  transactions  and  standard  text¬ 
editing  applications  that  will  reshape  daily  computing  and  have  a  significant  commercial  impact. 
To  realize  successful  multimodal  systems  for  the  future,  many  key  research  challenges  remain  to 
be  addressed.  Among  these  challenges  are  the  development  of  cognitive  theories  to  guide 
multimodal  system  design,  and  the  development  of  effective  natural  language  processing, 
dialogue  processing,  and  error-handling  techniques.  In  addition,  new  multimodal  systems  will  be 
needed  that  can  function  more  robustly  and  adaptively,  and  with  support  for  collaborative 
multiperson  use.  Before  this  new  class  of  systems  can  proliferate,  toolkits  also  will  be  needed  to 
promote  software  development  for  both  simulated  and  functioning  systems. 

[44]  Oviatt,  S.,  M.  MacEachern,  and  G.-A.  Levow.  “Predicting  Hyperarticulate  Speech  During 
Human-Computer  Error  Resolution.” 

When  speaking  to  interactive  systems,  people  sometimes  Hyperarticulate  -  or  adopt  a  clarified 
form  of  speech  that  has  been  associated  with  increased  recognition  errors.  The  goals  of  the 
present  study  were:  (1)  to  establish  a  flexible  simulation  method  for  studying  users’  reactions  to 
system  errors,  (2)  to  analyze  the  type  and  magnitude  of  linguistic  adaptations  in  speech  during 
human-computer  error  resolution,  (3)  to  provide  a  unified  theoretical  model  for  interpreting  and 
predicting  users’  spoken  adaptations  during  system  error  handling,  and  (4)  to  outline  the 
implications  for  developing  more  robust  interactive  systems.  A  semi-automatic  simulation 
method  with  a  novel  error  generation  capability  was  developed  to  compare  users’  speech 
immediately  before  and  after  system  recognition  errors,  and  under  conditions  varying  in  error 
base-rate.  Matched  original-repeat  utterance  pairs  then  were  analyzed  for  type  and  magnitude  of 
linguistic  adaptation.  When  resolving  errors  with  a  computer,  it  was  revealed  that  users  actively 
tailor  their  speech  along  a  spectrum  of  hyperarticulation,  and  as  a  predictable  reaction  to  their 
perception  of  the  computer  as  an  “at  risk”  listener.  During  both  lo  and  high  error  rates,  durational 
changes  were  pervasive,  including  elongation  of  the  speech  segment  and  large  relative  increases 
in  the  number  and  duration  of  pauses.  During  a  high  error  rate,  speech  also  was  adapted  to 
include  more  hyper-clear  phonological  features,  fewer  disfluencies,  and  change  in  fundamental 
frequency.  The  two-stage  CHAM  model  (Computer-elicited  Hyperarticulate  Adaptation  Model) 
is  proposed  to  account  for  these  changes  in  users’  speech  during  interactive  error  resolution. 
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[45]  Owei,  V.  (July  2002).  “An  Intelligent  Approach  to  Handling  Imperfect  Information  In  Concept- 
Based  Natural  Language  Queries.”  ACM  Transactions  on  Information  Systems  20:  291-328. 
Missing  information,  imprecision,  inconsistency,  vagueness,  uncertainty,  and  ignorance  abound 
in  information  systems.  Such  imperfection  is  a  fact  of  life  in  database  systems.  Although  these 
problems  are  widely  studied  in  relational  database  systems,  this  is  not  the  case  in  conceptual 
query  systems.  And  yet,  concept-based  query  languages  have  been  proposed  and  some  are 
already  commercial  products.  It  is  therefore  imperative  to  study  these  problems  in  concept-based 
query  languages,  with  a  view  to  prescribing  formal  approaches  to  dealing  with  the  problems.  In 
this  article,  the  author  has  done  just  that  for  a  concept-based  natural  language  query  system  that 
he  developed.  A  methodology  for  handling  and  resolving  each  type  of  imperfection  is  developed. 
The  proposed  approaches  are  automated  as  much  as  possible,  with  the  user  mainly  serving  an 
assistive  function. 

[46]  Quek,  F.,  D.  McNeill,  R.  Bryll,  S.  Duncan,  X.-F.  Ma,  C.  Kirbas,  K.E.  McCullough,  and  R. 

Ansari.  (September  2002).  “Multimodal  Human  Discourse:  Gesture  and  Speech.”  ACM 
Transactions  on  Computer-Human  Interaction  9:  I7I-I93. 

Gesture  and  speech  combine  to  form  a  rich  basis  for  human  conversational  interaction.  To 
exploit  these  modalities  in  HCI,  we  need  to  understand  the  interplay  between  them  and  the  way 
in  which  they  support  communication.  The  authors  propose  a  framework  for  the  gesture  research 
done  to  date,  and  present  their  work  on  the  cross-modal  cues  for  discourse  segmentation  in  free¬ 
form  gesticulation  accompanying  speech  in  natural  conversation  as  a  new  paradigm  for  such 
multimodal  interaction.  The  basis  for  this  integration  is  the  psycholinguistic  concept  of  the 
coequal  generation  of  gesture  and  speech  from  the  same  semantic  intent.  The  authors  present  a 
detailed  case  study  of  a  gesture  and  speech  elicitation  experiment  in  which  a  subject  describes 
her  living  space  to  an  interlocutor.  The  authors  perform  two  independent  sets  of  analyses  on  the 
video  and  audio  data:  video  and  audio  analysis  to  extract  segmentation  cues,  and  expert 
transcription  of  the  speech  and  gesture  data  by  microanalyzing  the  videotape  using  a  frame- 
accurate  videoplayer  to  correlate  the  speech  with  the  gestural  entities.  The  authors  compare  the 
results  of  both  analyses  to  identify  the  cues  accessible  in  the  gestural  and  audio  data  that 
correlate  well  with  the  expert  psycholinguistic  analysis.  The  authors  show  that  “handedness”  and 
the  kind  of  symmetry  in  two-handed  gestures  provide  effective  supersegmental  discourse  cues. 

[47]  Sun  Microsystems.  (October  26,  1998).  “Designing  Effective  Speech  Applications”  Chapter  3  in 
Java  Speech  API  Programmer’s  Guide, 

http://java.sun.com/products/java-media/speech/forDevelopers/jsapi-guide/UserInterface.html 
Speech  applications  are  like  conversations  between  the  user  and  the  computer.  Conversations  are 
characterized  by  turn-taking,  shifts  in  initiative,  and  verbal  and  non-verbal  feedback  to  indicate 
understanding.  A  major  benefit  of  incorporating  speech  in  an  application  is  that  speech  is  natural: 
people  find  speaking  easy,  conversation  is  a  skill  most  master  early  in  life  and  then  practice 
frequently.  At  a  deeper  level,  naturalness  refers  to  the  many  subtle  ways  people  cooperate  with 
one  another  to  ensure  successful  communication.  An  effective  speech  application  is  one  that 
simulates  some  of  these  core  aspects  of  human-human  conversation.  Since  language  use  is 
deeply  ingrained  in  human  behavior,  successful  speech  interfaces  should  be  based  on  an 
understanding  of  the  different  ways  that  people  use  language  to  communicate.  Speech 
applications  should  adopt  language  conventions  that  help  people  know  what  they  should  say  next 
and  that  avoid  conversational  patterns  that  violate  standards  of  polite,  cooperative  behavior.  This 
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chapter  discusses  when  a  speech  interface  is  and  is  not  appropriate,  and  then  provides  some 
concrete  design  ideas  for  creating  effective  speech  applications  that  adhere  to  conversational 
conventions. 

[48]  Ton,  L.,  and  Rothkrantz,  L.J.M.  (2001).  “Determining  User  Interface  Semantics  Using 
Communicating  Agents.”  TSD,  Springer- Verlag,  Berlin,  pgs.  349-356. 

The  Internet  offers  remote  access  to  many  information  systems  to  users  independent  of  time  and 
location.  This  paper  describes  an  agent  based  approach  to  deal  with  issues  that  rise  from  the 
difference  between  user  interfaces  to  functionally  similar  information  systems.  The  approach 
yields  agents  that  interact  with  user  interfaces  to  extract  the  model  of  the  underlying  information 
system.  The  model  is  represented  by  labeling  input  fields  of  the  interface  with  their  meaning. 
The  agents  are  modeled  after  human  dialogue  behavior  and  human  computer  interaction.  The 
interaction  agents  are  capable  of  independently  and  robustly  querying  the  information  system 
without  explicit  instructions.  Two  types  of  agents  are  implemented  and  tested  on  European  sites 
of  rail  travel  planners.  A  corpus  of  seven  rail  planners  was  created.  The  authors  modeled  human 
computer  interaction  based  on  the  analysis  of  these  planners.  A  notation  that  can  be  used  to 
describe  start  pages  based  on  their  functionality  was  defined.  A  framework  to  use  the  start  page 
description  language  to  fill  out  a  start  page  form  and  to  retrieve  the  results  was  created.  A  testing 
module  for  testing  result  pages  was  implemented.  Two  example  agents  for  extracting  user 
interface  semantics  was  developed.  These  agents  are  programmed  as  expert  systems  that  have 
rules  to  take  action  based  on  intermediate  and  result  pages.  A  testing  environment  was  created. 

[49]  Traum,  D.,  and  J.  Rickel.  (July  15-19,  2002).  ’’Embodied  Agents  for  Multi-party  Dialogue  in 
Immersive  Virtual  Worlds.”  AAMA5  ’02,  ACM:  766-773. 

Immersive  virtual  worlds  are  increasingly  being  used  for  education,  training,  and  entertainment, 
and  virtual  humans  that  can  interact  with  human  users  in  these  worlds  play  many  important  roles. 
However,  current  computational  models  of  dialogue  do  not  address  the  issues  that  arise  with 
face-to-face  communication  situated  in  three-dimensional  worlds,  such  as  the  proximity  and 
attentional  focus  of  others,  the  ability  to  maintain  multi-party  conversations,  and  the  interplay 
between  speech  and  nonverbal  signals.  This  paper  presents  a  new  model  that  integrates  and 
extends  prior  work  on  spoken  dialogue  and  embodied  conversational  agents,  and  describes  an 
initial  implementation  that  has  been  applied  to  training  in  virtual  reality. 

[50]  Wactlar,  H.D.,  and  C.-C.  Chen.  (July  13-17,  2002).  “Enhanced  Perspectives  for  Historical  and 

Cultural  Documentaries  Using  Informedia  Technologies.”  JCDL  ’02,  ACM:  338-339. 

Speech  recognition,  image  processing,  and  language  understanding  technologies  have 
successfully  been  applied  to  broadcast  news  corpora  to  automate  the  extraction  of  metadata  and 
make  use  of  it  in  building  effective  video  news  retrieval  interfaces.  This  paper  discusses  how 
these  technologies  can  be  adapted  to  cultural  documentaries  as  represented  by  awarding-winning 
Eirst  Emperor  of  China  videodisc  and  multimedia  CD.  Through  automated  means,  efficient 
interfaces  into  documentary  contents  can  be  build  dynamically  based  on  user  needs.  Such 
interfaces  enable  the  assemblage  of  large  video  documentary  libraries  from  component 
videodisc,  CD,  and  videotape  projects,  with  alternative  views  into  the  material  complementing 
the  original  sequences  authored  by  the  materials’  producers. 
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[51]  Walker,  M.A.,  DJ.  Litman,  C.A.  Kamm,  and  A.  Abella.  “PARADISE:  A  Framework  for 

Evaluating  Spoken  Dialogue  Agents.”  Ar<&r  Labs  -  Research,  pgs.  271-280. 

This  paper  presents  PARADISE  (PARAdigm  for  Dialogue  System  Evaluation),  a  general 
framework  for  evaluating  spoken  dialogue  agents.  The  framework  decouples  task  requirements 
from  an  agent’s  dialogue  behaviors,  supports  comparisons  among  dialogue  strategies,  enables  the 
calculation  of  performance  over  subdialogues  and  whole  dialogues,  specifies  the  relative 
contribution  of  various  factors  to  performance,  and  makes  it  possible  to  compare  agents 
performing  different  tasks  by  normalizing  for  task  complexity. 

[52]  Weinschenk,  S.,  and  D.T.  Barker.  (April  2000).  Designing  Effective  Speech  Interfaces.  John 

Wiley  &  Sons,  Inc. 

The  authors  focus  on  S/GUI  interfaces,  where  speech  is  part  of  a  multi-modal  interface,  as  well 
as  several  types  of  AUI  interfaces,  where  the  user  interacts  with  the  software  primarily  through 
speech.  They  clearly  explain  the  interface  design  principles  that  are  applied  to  speech  interfaces 
and  describe  the  latest  practices  of  leading  experts.  The  authors  also  provide  coverage  of  the 
special  issues  involved  in  interface  design  for  disabled  persons.  Along  with  its  in-depth  look  at 
speech  technologies  and  the  different  types  of  user  interfaces,  this  book:  (1)  Gives  an  overview 
of  the  field  of  human  factors  and  defines  the  basic  concepts  of  human  computer  interaction.  (2) 
Discusses  the  current  state  of  speech  technology  applications.  (3)  Explains  the  laws  of  human 
factors  that  apply  to  speech  interfaces.  (4)  Contains  guidelines  and  examples  for  user  control, 
human  limitation,  model  integrity  accommodation,  clear  dialogue,  and  aesthetic  integrity.  (5) 
Details  the  best  practices  in  interface  design  and  usability  engineering 

[53]  Yankelovich,  N.  (1997).  “Using  Natural  Dialogs  as  the  Basis  for  Speech  Interface  Design.” 
Chapter  in  Automated  Spoken  Dialog  Systems,  edited  by  S.  Euperfoy,  MIT  Press. 

In  sum,  this  experience  has  shown  that  natural  dialogs  can  serve  as  an  effective  starting  point  for 
a  speech  user  interface  design.  Not  only  do  they  help  in  the  design  of  grammars,  feedback,  and 
prompts,  but  they  also  point  out  instances  where  speech  technology  cannot  be  effectively 
applied.  These  case  studies  are  intended  to  demonstrate  that  extremely  useful  data  can  be 
collected  without  an  elaborate  setup  or  software  development  effort.  Both  the  Office  Monitor  and 
the  Drawing  studies  involved  minimal  preparation,  no  software  development,  and  less  than  an 
hour  of  recorded  data.  With  some  creative  thinking  about  the  study  design,  it  is  usually  easy  to 
construct  a  setting  for  interaction  that  approximates  the  interaction  that  will  occur  in  the  software 
system. 

[54]  Yankelovich,  N.,  and  K.  Ehrlich.  (November/December  1996).  “How  Do  Users  Know  What  To 

Say?”  ACM  Interactions  3:  pgs.  33-43. 
http://www.sun.com/research/speech/publications/ 

As  speech  recognition  technology  improves,  many  software  designers  are  being  challenged  to 
design  speech  user  interfaces.  For  the  author,  one  of  the  most  challenging  issues  is  “How  do 
users  know  what  they  can  say?”  A  speech-only  interface  poses  the  same  problems  as  a 
command-line  interface.  The  functionality  of  application  is  hidden,  and  the  boundaries  of  what 
can  and  cannot  be  done  are  invisible.  Graphical  interfaces  were  invented  largely  to  make  hidden 
functionality  visible  to  the  user.  In  a  speech-only  environment,  such  as  over  a  telephone,  it  is  not 
possible  to  display  menus,  show  options,  or  highlight  buttons.  Instead,  other  techniques  must  be 
used  to  guide  users  through  a  successful  interaction.  To  gather  insight  into  ways  of  tackling  this 
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difficult  problem,  the  author  asked  a  dozen  respected  speech  user  interface  designers  to  give  her 
examples  of  how  they  let  their  users  know  what  they  can  say.  The  paper  covers  the  following 
sections:  Defining  the  Problem,  Design  Constraints,  Techniques  (Explicit  Prompts,  Implicit 
Prompts,  Incremental  and  Expanded  Prompts,  Tapering,  Hints,  and  Techniques  Specific  to 
Multimodal  Systems).  User  experience  coupled  with  thoughtful  prompt  design  does  not 
guarantee  a  successful  speech  interface.  Other  design  issues  play  an  equally  important  role. 
These  include  crafting  grammars  or  command  languages,  providing  appropriate  feedback,  and 
recovering  from  errors.  In  addition,  a  consistent  “sound  and  feel”  across  speech  applications  can 
go  a  long  way  in  helping  users  know  what  to  say. 

[55]  Yankelovich,  N.,  G.-A.  Eevow,  and  M.  Marx.  (1995).  “Designing  SpeechActs:  Issues  in  Speech 
User  Interfaces.”  CHI  ’95  Proceedings. 

http://www.acm.org/sigchi/chi95/Electronic/documnts/papers/ny_bdy.htm 
SpeechActs  is  an  experimental  conversational  speech  system.  Experience  with  redesigning  the 
system  based  on  user  feedback  indicates  the  importance  of  adhering  to  conversional  conventions 
when  designing  speech  interfaces,  particularly  in  the  face  of  speech  recognition  errors.  Study 
results  also  suggest  that  speech-only  interfaces  should  be  designed  from  scratch  rather  than 
directly  translated  from  their  graphical  counterparts.  This  paper  examines  a  set  of  challenging 
issues  facing  speech  interface  designers  and  describes  approaches  to  address  some  of  those 
challenges.  Some  design  challenges  consist  of:  simulating  conversations,  pacing  of  the 
conversation,  transforming  GUIs  into  SUIs,  vocabulary,  organization  and  flow  of  information, 
recognition  /  rejection/  substitution  /  insertion  errors,  the  very  nature  of  speech  itself,  lack  of 
visual  feedback,  conversational  speech  and  persistence,  and  ambiguous  silence. 

[56]  Zhang,  Y.  (April  2000).  “Information  Eusion  for  Robust  Audio-Visual  Speech  Recognition.” 
Dissertation,  University  of  Illinois  at  Urbana-Champaign. 

Computer  technologies  have  improved  significantly  in  both  capacity  and  speed.  The  human- 
computer  interface  still  lags  behind,  in  that  human-computer  interaction  lacks  the  naturalness 
essential  for  efficient  communication.  In  order  to  make  human-computer  interaction  more 
natural,  novel  sensory  modalities  have  been  used.  Speech,  gestures,  and  emotional  states  can  be 
detected  and  understood  to  some  extent.  However,  integrating  these  information  sources  to 
achieve  performance  superior  to  any  single  modal  alone  remains  a  challenging  problem.  Humans 
naturally  and  effortlessly  perform  sensory  information  fusion.  The  most  important  human-to- 
human  communication  tool  is  speech.  Automatic  speech  recognition  by  machines  has  been  able 
to  achieve  very  high  recognition  accuracy  for  large  vocabulary  sets  and  speaker-independent 
tasks.  However,  in  some  environments,  unexpected  sources  of  noise  will  degrade  system 
performance.  The  author  proposes  a  novel  integration  technique  that  can  efficiently  incorporate 
both  visual  and  acoustic  speech  signals  to  achieve  better  speech  recognition  accuracy  than  that 
achieved  by  either  of  the  single  modalities  alone.  The  proposed  fusion  schemes  have  been  tested 
in  different  situations.  The  experiment  results  show  consistently  improved  performance  by  using 
multimodal  fusion  schemes. 
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e.  Typography 

[57]  Black,  A.  and  Boag,  A.  (1992)  Choosing  Binary  or  Greyscale  Bitmaps:  some  Consequences  for 
Users.  EP92,  Cambridge  University  Press:  247-260. 

Two  studies  investigated  using  binary  or  greyscale  bitmap  fonts  in  reading,  proofreading  with 
correction,  and  identifying  typefaces.  At  a  fixed  size  (10  pt)  ease  of  reading,  and  proofreading, 
did  not  vary  according  to  font  type  (binary  or  greyscale),  but  according  to  apparent  size  of  the 
typefaces  tested.  Typeface  identification  was  significantly  more  accurate  with  greyscale  bitmaps 
than  with  binary  bitmaps.  We  conclude  that  in  contexts  where  legibility  is  essential,  such  as  in 
electronic  texts,  greyscale  bitmaps  are  subject  to  the  same  sorts  of  constraints  as  binary  bitmaps, 
and  typeface  and  size  must  be  chosen  carefully.  However  greyscaling  is  likely  to  improve  the 
user  interface  for  tasks  such  as  designing  paper  documents  on  screen,  where  the  accurate 
reproduction  of  letterforms  is  important.  Further  study  should  demonstrate  which  screen-based 
design  decisions  are  supported  well  by  greyscale  representations. 

[58]  Boyarski,  D.  Neuwirth,  C.  Forlizzi,  J.  and  Harkness  Regli,  S.  (1998)  “A  Study  of  Fonts  Designed 
for  Screen  Display.”  CHI  ‘98  Papers:  87-93. 

This  study  examined  the  readability  and  subjective  preferences  of  a  set  of  fonts  designed  for 
screen  display.  Two  new  binary  bitmap  fonts  performed  well,  suggesting  that  designers  should 
consider  incorporating  similar  attributes  into  default  fonts  for  on-line  type. 


[59] ]  Duesbury,  R.  T.  and  O’Neil,  Jr.  H.F.  (1996)  “Effect  of  Type  of  Practice  in  a  Computer-Aided 

Design  Environment  in  Visualizing  Three-Dimensional  Objects  From  Two-Dimensional 
Orthographic  Projections.”  Journal  Of  Applied  Psychology.  81(3):  249-260. 

The  purpose  of  this  study  was  to  determine  the  effect  of  practice  in  manipulating  2-  and  3- 
dimensional  (D)  wireframe  images  on  a  learner’s  ability  to  visualize  3-D  objects.  Practice,  either 
rotational  or  not,  consisted  of  visualizing  2-  and  3-D  objects  generated  by  personal  computer 
(PC)-based  computer-assisted  design  software.  Results  indicated  that  participants  in  the  rotation 
treatment  group  performed  significantly  better  than  those  in  either  the  nonrotation  or  control 
group  on  measures  of  spatial  ability  and  3-D  visualization  ability.  Both  treatment  groups 
performed  significantly  better  than  the  control  group  on  measures  of  metacognition,  effort,  and 
worry.  These  results  support  a  conclusion  that  spatial  ability  can  be  improved  through  practice 
that  allows  the  learner  to  see  the  relationship  between  the  2-D  and  3-D  features  of  objects. 

[60]  Dumbeck,  K.  (1999)  “Book  Review:  Typography  on  the  Web.”  Technical  Communication, 
Second  Quarter:  263-265. 

As  if  keeping  ourselves  educated  on  the  rules  of  typography  (if  there  are  indeed  actual  rules)  for 
the  print  media  hasn’t  kept  us  busy  enough,  we  must  now  begin  the  serious  consideration  of 
typography  as  it  pertains  to  Web  publications.  While  this  may  seem  straightforward  to  some,  on 
further  consideration,  you  may  find  there  is  more  to  it  than  meets  the  naked,  paper- trained  eye. 
This  book  does  an  excellent  job  of  providing  a  fairly  comprehensive  overview  of  many  of  the 
issues  that  designers  and  typographers  must  consider  when  creating  Web  publications.  The  book 
is  divided  into  the  following  seven  parts,  moving  logically  from  fundamental  to  more  advanced 
topics:  (1)  Fundamentals  of  typography,  (2)  Digital  typography,  (3)  Typography  on  the  Web,  (4) 
Cascading  Style  Sheets,  (5)  TrueDoc,  (6)  Font  embedding,  and  (7)  Advanced  topics  in  Web 
typography. 
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[61]  Durand,  D.  G.  DeRose,  SJ.  and  Mylonas,  E.  What  Should  Markup  Really  Be?  Applying 
Theories  of  Text  to  the  Design  of  Markup  Systems. 
http:://www. cs.bu.edu/students/grads/dgd/markup_abridged.htm 

The  issue  of  what  text  really  is,  and  how  it  affects  our  notions  of  proper  text  representation  has 
been  with  us  almost  from  the  beginning  of  text  encoding  [Goldbarb  1981,  Reid  1980,  Coombs,  et 
al.  1987,  DeRose,  et  al.  1990,  Renear  et  al.].  The  simplest  reasonable  view,  that  text  is 
fundamentally  an  ordered  hierarchical  structure,  determined  by  its  editor  and  author,  is  an  early 
one  that  has  remained  prominent,  especially  as  reified  by  ISO  8879  (SGML).  However,  this 
simple  model  is  not  enough,  which  the  TEI  [Sperberg-McQueen  and  Burnard  1990,  1993] 
quickly  discovered  as  it  moved  text  encoding  from  the  realm  of  print  production  to  that  of 
scholarship,  textual  editing,  and  linguistic  analysis.  The  TEI  metalanguage  committee  identified 
problems  with  SGML’s  simple  hierarchical  mechanisms,  and  developed  and  published 
techniques  for  working  around  them  to  encode  non-hierarchical  phenomena  [Barnard  et  al, 
1996]. 


[62]  Gilreath,  C.  T.  (1993)  “Graphic  Cueing  of  Text:  The  Typographic  and  Diagraphic  Dimensions.” 
Visible  Language  27(3):  336-361. 

A  new  taxonomy  is  proposed  for  classifying  the  graphic  cues  commonly  used  in  visually 
informative  text.  Previous  approaches  have  focused  on  typographic  and  spatial  cueing  but  have 
not  formalized  the  concept  which  the  author  calls  mark  cueing.  Mark  cues  are  lines  such  as 
dividers,  guidelines  and  network  links  and  visual  tags  such  as  bullets  and  enumerators.  Spatial 
and  mark  cueing  are  subsumed  under  a  new  concept  called  diagraphic  cueing.  Together, 
diagraphic  and  typographic  cueing  make  up  the  broader  concept  of  graphic  cueing.  The  various 
forms  of  graphic  cues  are  surveyed,  with  a  brief  look  at  products  such  as  vertical  lists,  tables, 
network  diagrams  and  text  labels. 


[63]  Golovchinsky,  G.  Kamps,  T.  and  Reichenberger,  K.  (1995)  “Subverting  Structure:  Data-driven 
Diagram  Generation.”  IEEE  Conference  Proceedings  on  Visualization  ‘95:  217-223,  455.  ISBN: 
0-8186-7187-4,  IEEE  Catalog  Number:  95CB35835. 

Diagrams  are  data  representations  that  convey  information  predominantly  through  combinations 
of  graphical  elements  rather  than  through  other  channels  such  as  text  or  interaction.  The  authors 
have  implemented  a  prototype  called  AVE  (Automatic  Visualization  Environment)  that 
generates  diagrams  automatically  based  on  a  generative  theory  of  diagram  design.  According  to 
this  theory,  diagrams  are  constructed  based  on  the  data  to  be  visualized  rather  than  by  selection 
from  a  predefined  set  of  diagrams.  This  approach  can  be  applied  to  knowledge  represented  by 
semantic  networks.  In  this  paper  they  give  a  brief  introduction  to  the  underlying  theory,  then 
describe  the  implementation  and  finally  discuss  strategies  for  extending  the  algorithm. 


[64]  Ishizaki,  S.  and  Lokuge,  I.  (1995)  Intelligent  Interactive  Dynamic  Maps.  Proceedings  of 
AutoCarto  12.  MIT  Media  Laboratory’s  Visible  Language  Workshop.  Sponsored  in  part  by 
ARPA,  JNIDS,  NYNEX  and  Alenia. 

This  paper  presents  an  experimental  intelligent  map  system  -  GeoSpace  -  which  allows 
information  seekers  to  explore  complex  spaces  of  geographic  information  using  dialogue-like 
interaction.  GeoSpace  progressively  and  selectively  provides  information  as  an  information 
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seeker  enters  queries  while  visually  maintaining  the  larger  context.  Domain  knowledge  is 
represented  in  a  form  of  information  presentation  plan  modules,  and  an  activation  spreading 
network  technique  is  used  to  determine  the  relevance  of  information  based  on  information 
seeking  queries.  The  reactive  nature  of  the  activation  spreading  network,  combined  with  visual 
design  techniques,  such  as  typography,  color,  and  transparency,  enables  the  system  to  fluidly 
respond  to  the  continuous  changes  in  the  information  seeker’s  goals  and  intentions. 


[65]  Ishizaki,  S.  (1994)  Adjusting  Simultaneous  Contrast  Effect  for  Dynamic  Information  Display. 
IS&T  and  SID’s  Color  Imaging  Conference:  Color  Science,  Systems  and  Applications. 
http://www.andrew.cmu.edu/user/suguru/me/publications.html 

Simultaneous  contrast  effect  often  confuses  the  viewer’s  understanding  of  color  coding  scheme 
on  information  graphics,  such  as  maps  and  diagrams.  In  particular,  on  a  computer-based  dynamic 
display,  such  as  weather  and  air  traffic,  since  background  color  and  position  of  a  graphical 
element  are  difficult  to  predict  at  run-time,  we  need  to  adjust  the  physical  color  of  each  element 
automatically  so  that  all  elements  that  are  intended  to  appear  the  same  color  are  perceived  that 
way.  This  paper  introduces  experimental  information  graphics  that  automatically  adjusts  color 
differences  based  on  Jameson  and  Hurvich’s  research.  The  results  suggest  that  adjustment  of 
simultaneous  contrast  effect  can  increase  the  reliability  of  dynamic  information  display  and  the 
flexibility  of  its  design. 


[66]  Jansen,  C.  (1994)  “Research  in  Technical  Communication  in  the  Netherlands.”  Technical 
Communication  41(2):  234-239. 

In  the  Netherlands,  research  in  technical  communication  is  a  part  of  research  in  “functional  text,” 
which  has  concrete  goals  that  must  be  achieved  by  lay  readers.  Three  recent  studies  focus  on  the 
use  of  and  failure  to  use  software  manuals,  the  minimalist  approach  and  learning  styles,  and  the 
effect  of  using  decision  tables. 


[67]  Keyes,  E.  (1993)  “Typography,  Color,  and  Information  Structure.”  Technical  Communication, 
Fourth  Quarter:  638-654. 

Typography  and  color  significantly  influence  the  communication  effectiveness  of  technical 
documents.  However,  most  research  and  design  guidelines  address  typography  and  color 
separately  without  considering  their  spatial  context  or  their  function  as  complementary  content 
signals.  A  broader,  integrated  perspective  increases  our  understanding  of  underlying  principles, 
simplifies  guidelines,  and  enables  us  to  make  typography  and  color  choices  with  confidence  and 
control  over  the  result.  This  articles  focuses  on:  (1)  How  typography  and  color  complement  and 
differ  from  each  other  in  signaling  an  underlying  content  structure,  (2)  The  synergism  between 
typography,  color,  and  page  layout  (use  of  white  space)  that  aids  audience  understanding  and 
use,  and  (3)  The  characteristics  of  typography  and  of  color  that  are  most  important  in  these 
contexts. 


[68]  Microsoft  (1995)  “Displaying  Text  on  NTSC:  A  Research  Summary.”  10  pgs. 

This  document  outlines  the  kinds  of  projects  which  will  use  NTSC  font  technology.  It  describes 
the  basic  problems  associated  with  the  display  of  text  on  TVs  using  the  NTSC  signal,  and 
proposes  three  possible  approaches  towards  a  solution  of  these  problems.  It  also  includes  an 
annotated  source  of  references,  and  a  list  of  further  articles  which  may  prove  useful,  but  have  not 
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yet  been  unearthed.  The  text  of  the  document  draws  (fairly  heavily  at  times)  on  the  words  of 
Charles  Bigelow  and  Will  Adams  (among  others)  -  its  chief  virtue  being  in  the  pulling  together 
of  as  many  separate  sources  as  could  be  found  in  the  research  period.  Finding  fonts  and 
techniques  to  provide  a  legible  display  on  a  TV  or  similar  device  using  an  NTSC  signal  as  its 
input  will  allow  Windows  and  other  software  to  be  run  on  a  computer  using  a  TV  as  its  display 
device.  As  discussed,  the  difficulties  of  font  design  and  employment  for  the  TV  are  essentially 
very  different  from  those  facing  the  designer  and  user  of  screen  fonts  for  the  computer  monitor. 
Projects  which  will  use  NTSC  font  technology  include  low  cost  PCs  using  a  TV  as  a  display 
device;  set-top  boxes  for  interaction  TV;  applications  such  as  PowerPoint  or  video  titling  that 
output  directly  to  video  tape. 


[69]  Spyridakis,  J.  H.,  and  Wenger,  M.J.  (1992).  “Writing  for  Human  Performance:  Relating  Reading 
Research  to  Document  Design.”  Technical  Communication,  Second  Quarter:  202-215. 

This  paper  briefly  reviews  a  few  models  of  reading  performance  as  a  point  of  departure  and  as  an 
organizing  framework  for  understanding  findings  from  empirical  studies  of  text  and  reader 
factors.  It  then  reviews  numerous  empirical  studies  of  text  design  and  reader  variables  and  their 
effects  on  comprehension.  Its  goal  is  to  help  readers  understand  this  research  and  identify  the 
implications  of  such  research  in  document  design  decisions. 


[70]  Van  Der  Waarde,  K.  (1999)  “Typographic  Dimensions  and  Conventional  Wisdom:  A 
Discrepancy?”  Technical  Communication,  First  Quarter:  67-74. 

This  article  finds  the  x-height,  line  length,  and  line  spacing  in  brochures,  scientific  journals,  and 
novels  do  not  reflect  typographic  guidelines.  Also,  the  article  demonstrates  that  the  boundaries 
for  these  typographic  dimensions  differ  by  genre. 


[71]  Wendt,  D.  (August  1992)  “Legibility  of  Print.”  Department  of  Psychology,  University  of  Kiel, 
Germany,  18  pgs. 

What  do  we  mean  by  legibility?  The  main  purpose  of  a  type  face  is  the  transfer  of  information. 
To  reach  this  goal,  it  has  to  be  legible,  i.e.  the  recipient  of  the  information  should  be  able  to 
decode  it  with  as  little  effort  as  possible,  and  re-assess  the  symbols  to  the  objects  and  processes 
they  represent.  This  is  generally  agreed  upon,  but  it  is  more  difficult  to  agree  upon  how  this 
legibility  can  be  assessed,  or  even  measured. 


[72]  Wilkinson,  R.  T.  and  Robinshaw,  H.M.  (1987)  “Proof-Reading:  VDU  and  Paper  Text  Compared 
for  Speed,  Acuracy,  and  Fatigue.”  Behaviour  and  Information  Technology  6(2):  125-133. 
Proof-reading  on  either  a  cathode  ray  tube  visual  display  unit  (VDU)  of  average  quality  or  on 
conventional  paper  typescript  was  carried  out  for  four  50-min  sessions,  two  in  each  condition. 
The  method  of  signaling  errors  in  the  script  was  identical  in  each  case:  verbally  and  by  line 
reference.  VDU  scripts  were  presented  in  a  standard  Wordstar  format;  paper  text  was  a  normal 
print-out  of  that  script.  With  the  VDU  as  compared  to  paper,  more  proofreading  errors  were 
missed,  fewer  pages  were  read,  and  there  was  a  greater  accumulation  of  fatigue  during  the 
reading  session  as  indexed  by  an  increase  in  the  number  of  errors  missed.  It  is  suggested  that  (i) 
material  be  printed  for  proof-reading,  and  (ii)  the  present  technique  be  used  for  comparison  of 
different  VDUs  for  speed  and  accuracy  of  reading  and  resistance  to  fatigue. 
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[73]  Wise,  D.  (1999)  “The  Need  to  Know;  “Trivial  Pursuit”;  Conventional  Wisdom  about 
Typography.”  Technical  Communication,  Third  Quarter:  309-314. 

Nine  pieces  of  correspondence  regarding  articles  and  reviews  previously  published  in  the 
journal. 

f.  Usability  and  Evaluation 

[74]  Barnett,  M.  (1998)  “Testing  a  Digital  Library  of  Technical  Manuals.”  IEEE  Transactions  on 
Professional  Communication  41(2):  116-122. 

This  paper  describes  a  methodology  for  testing  the  usability  of  a  digital  library.  The  paper  also 
presents  the  results  from  using  this  methodology  on  a  specific  library  of  technical  manuals.  The 
testing  process  involves  timing  subjects  while  they  look  up  facts  in  comparable  libraries  of 
online  and  paper  documents.  They  are  timed  for  both  how  long  they  take  to  find  a  desired 
manual  in  the  library  and  how  long  they  take  to  find  a  desired  fact  in  a  chosen  manual.  Next,  the 
subjects  fill  out  a  questionnaire  on  which  type  library  they  prefer  using  and  why.  The  objective 
time  results  and  the  subjective  preferences  are  compared  and  analyzed. 

[75]  Bias,  R.  G.  and  Mayhew,  D.J.  (1994)  Cost-Justifying  Usability,  Academic  Press. 

This  book  provides  structured  and  proven  techniques  for  usability  engineering  professionals  and 
their  managers  for  quantifying  costs  and  benefits  in  order  to  make  a  convincing  and  successful 
business  case  for  investment.  The  book  provides  a  universal  framework  for  cost  justification;  an 
examination  of  factors  unique  to  vendor  companies,  internal  development  organizations,  and 
contractors;  case  studies  of  successful  cost  justification  efforts;  and  an  overview  of  discount 
usability  techniques,  use  of  specialized  tools  for  cost  justification,  and  a  look  toward  the  future  of 
usability  engineering. 

[76]  Boy,  G.  A.  (1998).  Cognitive  Eunction  Analysis  for  Human-Centered  Automation  of  Safety- 
Critical  Systems.  CHI  ‘98  Papers,  pgs.  265-272. 

The  Cognitive  Eunction  Analysis  is  a  methodology  supported  by  a  mediating  tool  for  the  human- 
centered  automation  of  safety-critical  systems  [4] .  It  is  based  on  a  socio-cognitive  model  linking 
the  artifact  being  designed,  the  user’s  activity,  the  task  to  be  performed,  and  the  organizational 
environment.  Cognitive  functions  can  be  allocated  to  humans  or  machines.  They  are 
characterized  by  their  role,  context  definition  and  associated  resources.  The  methodology  is 
supported  by  active  design  documents  as  mediating  representations  of  the  artifact,  the  interaction 
description  and  cognitive  function  descriptors  being  designed,  redesigned,  and  used  as  usability 
criteria  to  evaluate  the  distribution  of  cognitive  functions  among  humans  and  machines.  This 
methodology  enhances  user-centered  and  participatory  design,  and  traceability  of  design 
decisions.  It  was  successfully  tested  on  three  main  applications  in  the  aeronautics  domain.  One  of 
them  is  presented. 
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[77]  Carliner,  S.  (1997)  “Demonstrating  Effectiveness  and  Value:  A  Process  for  Evaluating  Technical 
Communication  Products  and  Services.”  Technical  Communication,  Third  Quarter:  252-265. 

Just  like  the  North  American  explorers  seeking  the  elusive  Northwest  Passage,  we  technical 
communicators  have  been  looking  for  a  certain  statistic  that,  if  we  were  to  find  it  and  measure  it, 
could  prove  without  a  doubt  that  our  work  was  of  high  quality  and  that  would  demonstrate, 
without  question,  the  value  of  a  technical  communicator’s  services  to  an  employer.  We  devised 
many  approaches  to  assessing  quality  and  demonstrating  value  beginning  in  the  1970s  and 
1980s.  They  include:  (1)  Counting  comments  on  drafts,  hoping  to  establish  a  correlation  between 
the  number  of  review  comments  and  the  quality  of  communication  products,  (2)  Assessing 
conformance  to  requirements  (Bandes,  1986).  This  approach  reflects  the  industrial  definition  of 
quality  because  it  is  the  same  approach  used  to  assess  the  quality  of  manufactured  parts,  like 
circuit  boards,  and  (3)  Devising  lists  of  characteristics  associated  with  quality  communication 
products  (such  as  the  U-metric  devised  by  Hewlett-Packard).  Since  the  early  1990s,  we  have 
generally  concluded  that  there  is  no  single,  unquestionable  measure  of  quality  and  value.  We 
reached  that  conclusion,  in  part,  because  we  are  increasingly  distinguishing  between  the  terms 
quality  and  value,  and  recognize  that  they  are  not  synonymous. 


[78]  Dumas,  J.  and  Redish,  J.  (1999).  A  Practical  Guide  to  Usability  Testing  (Revised  Edition), 
Intellect  Etd. 

In  this  book,  the  authors  begin  by  defining  usability,  advocating  and  explaining  the  methods  of 
usability  engineering  and  reviewing  many  techniques  for  assessing  and  assuring  usability 
throughout  the  development  process.  The  authors  then  take  you  through  all  the  steps  in  planning 
and  conducting  a  usability  test,  analyzing  data,  and  using  the  results  to  improve  both  products 
and  processes.  This  book  discusses  the  full  range  of  testing  options  from  quick  studies  with  a  few 
subjects  to  more  formal  tests  with  carefully  designed  controls.  The  authors  discuss  the  place  of 
usability  laboratories  in  testing  as  well  as  the  skills  you  need  to  conduct  a  test.  Included  are 
forms  that  you  can  use  or  modify  to  conduct  a  usability  test  and  layouts  of  existing  labs  that  will 
help  you  build  your  own. 

[79]  Houser,  S.A.,  Neisler,  R.P.,  and  Belcher,  G.J.  (November  2000).  “Interactive  Electronic 
Technical  Manual  Cost-Benefit  Analysis  Tool  -  User’s  Guide.”  Logistics  Management  Institute, 
McEean,  VA. 

This  report  is  the  user’s  guide  for  the  Interactive  Electronic  Technical  Manual  Cost-Benefit  Tool. 
It  provides  a  description  of  the  tool’s  layout  and  several  tutorials  that  instruct  the  user  how  to 
operate  the  tool.  The  lETM  Cost-Benefit  Analysis  Tool  is  an  Excel  spreadsheet-based  model  that 
compares  the  costs  and  benefits  of  candidate  lETM  deployment  alternatives.  The  tool  applies 
technology  from  previous  EMI  cost-benefit  models  for  law  enforcement  and  commercial 
aviation  to  lETMs.  The  current  model  has  three  modes  of  operation:  standard  analysis, 
sensitivity  analysis,  and  simulation.  The  tool  has  a  utility  worksheet  that  helps  the  user  to 
estimate  the  cost  of  converting  a  paper  manual,  or  a  low-level  electronic  manual  (ETM),  to  a 
higher-level  ETM  or  lETM. 

[80]  Ivory,  M.Y.,  and  Hearst,  M.A.  (December  2001).  “The  State  of  the  Art  in  Automating  Usability 

Evaluation  of  User  Interfaces.”  ACM  Computing  Surveys  33:  470-516. 

Usability  evaluation  is  an  increasingly  important  part  of  the  user  interface  design  process. 
However,  usability  evaluation  can  be  expensive  in  terms  of  time  and  human  resources,  and 
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automation  is  therefore  a  promising  way  to  augment  existing  approaches.  This  article  presents  an 
extensive  survey  of  usability  evaluation  methods,  organized  according  to  a  new  taxonomy  that 
emphasizes  the  role  of  automation.  The  survey  analyzes  existing  techniques,  identifies  which 
aspects  of  usability  evaluation  automation  are  likely  to  be  of  use  in  future  research,  and  suggests 
new  ways  to  expand  existing  approaches  to  better  support  usability  evaluation. 


[81]  Landauer,  T.  K.  (1995).  The  Trouble  With  Computers:  Usefulness,  Usability,  and  Productivity. 
Cambridge,  Mass.  MIT  Press. 

This  book  pieces  together  a  picture  of  a  situation,  of  how  and  where  computers  and  related 
information  technologies  have  failed  to  fulfill  their  promise  to  be  helpful.  Actually,  this  book  is 
not  about  just  computers  but  about  the  whole  range  of  electronic  technologies  used  in  business. 
Many  examples  and  data  the  authors  look  at  concern  IT,  or  information  technology  -  a  grab-bag 
category  of  computers,  telecommunications  equipment,  factory  automation  gadgets,  and  office 
gear.  In  dollars,  computers  are  a  very  big  part  of  this  category,  and  many  of  the  other  things  in  it 
that  don’t  have  screens  and  flashing  lights  either  contain  hidden  computers  (copiers  and  cash 
registers)  or  rely  on  close  friendships  with  computers  (telephones  and  MRIs).  It’s  this  mix  of 
stuff  -  computer-reliant  or  computer- associated  information  technology  -  that  seems  to  be  doing 
poorly.  The  author  offers  an  analysis  and  diagnosis  of  why  computers  and  friends  are  not  doing 
better  than  they  are  and  suggest  a  set  of  remedies.  Finally,  the  author  speculates  a  bit  on  what  the 
world  will  be  like  when  computer  power  is  finally  properly  harnessed. 

[82]  Lumsden,  J.,  and  Gray,  P.  (2001).  “SUIT  -  Context  Sensitive  Evaluation  of  User  Interface 
Development  Tools.”  DSV-IS,  Springer- Verlag,  Berlin,  pgs.  79-95. 

Developers  of  interactive  software  are  confronted  by  a  variety  of  software  tools  to  help  them 
design  and  implement  user  interfaces.  They  often  resort  to  ad  hoc  means  of  tool  selection  and 
subsequently  are  dissatisfied  with  their  chosen  tool.  In  this  paper  the  authors  describe  a 
framework,  evaluation  methodology,  and  associated  tool  for  investigating  the  suitability  of  user 
interface  development  tools  (UIDTs)  for  use  in  software  development  organizations  and  projects. 
The  authors  also  present  the  results  of  two  informal  empirical  studies  carried  out  in  support  of 
the  development  of  their  framework  and  method. 


[83]  Nielsen,  J.  and  Molich,  R.  (1990).  Heuristic  Evaluation  of  User  Interfaces.  ACM  CHI  ‘90 
Conference  Proceedings,  Seattle,  Washington,  pgs.  249-256. 

Heuristic  evaluation  is  an  informal  method  of  usability  analysis  where  a  number  of  evaluators  are 
presented  with  an  interface  design  and  asked  to  comment  on  it.  Pour  experiments  showed  that 
individual  evaluators  were  mostly  quite  bad  at  doing  such  heuristic  evaluations  and  that  they 
only  found  between  20  and  51%  of  the  usability  problems  in  the  interfaces  they  evaluated.  On 
the  other  hand,  we  could  aggregate  the  evaluations  from  several  evaluators  to  a  single  evaluation 
and  such  aggregates  do  rather  well,  even  when  they  consist  of  only  three  to  five  people. 


[84]  Nielson,  J.  and  Mack,  R.E.  (1994).  Usability  Inspection  Methods,  John  Wiley  and  Sons,  Inc. 

This  book  is  designed  to  get  you  quickly  up  and  running  with  the  full  complement  of  Usability 
Interface  (UI)  strategies,  tools,  and  techniques.  With  the  help  of  numerous  real-life  case  studies, 
the  authors  provide:  (1)  Step-by-step  guidance  on  all  important  methods  now  in  use,  including 
the  heuristic  evaluation  method,  the  pluralistic  walkthrough  method,  and  the  cognitive 
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walkthrough  method;  (2)  Proven  techniques  for  integrating  usability  inspections  with  other 
methods  now  in  use;  (3)  An  in-depth,  comparative  analysis  of  UI  versus  user  testing;  (4)  A  cost- 
benefit  analysis  of  US  as  compared  to  other  approaches;  (5)  Program  prototypes  that  provide  UI 
computer  support  for  interface  designers;  and  (6)  An  important  resource  for  user  interface 
developers,  software  designers,  as  well  as  graduate  students  and  researchers. 

[85]  Paterno,  F.,  and  Santoro,  C.  (2001).  “Integrating  Model  Checking  and  HCI  Tools  to  Help 

Designers  Verify  User  Interface  Properties.”  DSV-IS,  Springer- Verlag,  Berlin,  pgs.  135-150. 

In  this  paper  the  authors  present  a  method  that  aims  to  integrate  the  use  of  formal  techniques  in 
the  design  process  of  interactive  applications,  with  particular  attention  to  those  applications 
where  both  usability  and  safety  are  main  concerns.  The  method  is  supported  by  a  set  of  tools. 
The  authors  discuss  how  the  resulting  environment  can  be  helpful  in  reasoning  about  multi-user 
interactions  using  the  task  model  of  an  interactive  application.  Examples  are  provided  from  a 
case  study  in  the  field  of  air  traffic  control. 


[86]  Preece,  J.  (1993).  A  Guide  to  Usability:  Human  Factors  in  Computing.  Addison-Wesley. 

A  revolution  in  usability  is  fueling  the  continued  growth  of  the  computer  industry.  Graphical 
user  interfaces  and  improved  input/output  devices  have  helped  non-specialists  to  master  the 
awesome  power  and  complexity  of  the  modem  computer.  Despite  these  advances,  there  is  still  a 
need  to  raise  awareness  of  the  human  factors  in  user  interface  design  and  this  book  is  designed  to 
meet  that  need.  This  book  presents:  (1)  A  clear,  concise  account  of  human  factors  in  computing, 
illustrated  with  numerous  examples,  figures  and  cartoons,  (2)  An  excellent  balance  between  the 
technical  and  psychological  issues  in  human-computer  interaction  (HCI),  and  (3)  An  extensive 
glossary  and  suggestions  for  further  reading. 

[87]  Siewiorek,  D.,  A.  Smailagic,  and  M.  Hornyak.  (October  14-16,  2002).  “Multimodal  Contextual 
Car-Driver  Interface.”  Fourth  IEEE  ICMl  ’02  -  International  Conference  on  Multimodal 
Interfaces,  Pittsburgh,  PA,  pgs.  367-373. 

This  paper  focuses  on  the  design  and  implementation  of  a  Companion  Contextual  Car  Driver 
interface  that  proactively  assists  the  driver  in  managing  information  and  communication.  The 
prototype  combines  a  smart  car  environment  and  driver  state  monitoring,  incorporating  a  wide 
range  of  input-output  modalities  and  a  display  hierarchy.  Intelligent  agents  link  information  from 
many  contexts,  such  as  location  and  schedule,  and  transparently  learn  from  the  driver,  interacting 
with  the  driver  only  when  it  is  necessary. 


[88]  Tinker,  M.  A.  (1955).  Examiner’s  Manual  for  Tinker  Speed  of  Reading  Test.  Published  by  the 
University  of  Minnesota  Press,  Minneapolis,  Minnesota. 

This  test  was  constructed  to  provide,  first,  a  measure  of  speed  of  reading  as  a  single  variable 
uncomplicated  by  comprehension  difficulties,  i.e.  speed  of  reading  with  comprehension  held 
constant;  and,  second,  a  performance  test  which  may  be  employed  with  any  working  time  up  to 
30  minutes.  This  test  has  yielded  valid  and  reliable  speed  of  reading  scores  for  a  variety  of  time 
limits  from  4  to  30  minutes  on  each  form.  The  materials  in  the  test  are  such  that  it  may  be  used 
with  high  school  and  adult  subjects.  The  test  was  developed  for  two  main  uses:  (1)  For  carrying 
out  research  on  visual  functions,  such  as  the  effect  of  typographical  variations  on  the  speed  of 
perception  in  reading  or  the  effect  of  illumination  on  the  speed  of  perception  in  reading.  (2)  For 
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measuring  speed  of  sustained  reading  in  situations  where  performance  over  a  relatively  long 
period  of  time  is  desirable,  as  in  certain  counseling  setups  in  colleges  and  high  schools. 


g.  User  Interface  Design  and  Navigation 

[89]  Amau,  R.C.,  Thompson,  R.L.,  and  Cook,  C.  (February  2001).  “Do  Different  Response  Formats 
Change  the  Latent  Structure  of  Responses?  An  Empirical  Investigation  Using  Taxometric 
Analysis.”  Educational  and  Psychological  Measurement,  61:  23-44. 

Although  Web-based  surveys  are  increasing  in  popularity,  very  little  research  has  been 
conducted  on  the  psychometric  implications  of  using  different  user  interfaces  for  eliciting 
responses  to  survey  items.  The  purpose  of  this  study  was  to  compare  the  latent  structures  of 
responses  to  two  different  user  interface  response  formats  in  a  Web-based  survey.  Two  different 
coherent  cut  kinetics  Taxometric  procedures  -  M AMBAC  and  L-mode  factor  analysis  -  were 
used  to  examine  the  latent  structure  of  responses  to  a  survey  of  library  service  quality  using  an 
unnumbered  slider-bar  user  interface  versus  a  radio-button  user  interface.  Strong  evidence  was 
found  for  a  pure  dimensional  latent  structure  of  responses  from  both  user  interfaces,  which 
replicated  across  both  Taxometric  procedures  and  across  multiple  survey  items.  It  is  concluded 
that  the  slider-bar  and  radio-button  user  interfaces  both  yield  similar  latent  structures  of  survey 
item  responses.  These  analyses,  conducted  on  a  very  large  data  set,  offer  important  information 
for  Web-based  survey  designers  because,  although  slider  bars  are  more  visually  attractive,  they 
require  more  sophisticated  hardware  and  software.  These  findings  suggest  that  radio  buttons  can 
be  used  instead  of  Java-based  slider  bars  as  a  “lowest  common  denominator”  that  can  be 
expected  to  work  on  almost  any  Web  browser,  potentially  leading  to  higher  response  rates. 


[90]  Bass,  L.,  Kasabach,  C.,  Martin,  R.,  Siewiorek,  D.,  Smailagic,  A.,  Stivoric,  J.  (1997)  “The  Design 
of  a  Wearable  Computer.”  ACM  CHI  ’97  Conference  Proceedings  on  Human  Factors  in 
Computing  Systems,  Atlanta,  GA.  pgs.  139-146. 
http://www.acm.org/pubs/citations/proceedings/chi/258549/pl39-bass/ 

The  design  process  used  to  produce  an  innovative  computer  system  is  presented.  The  computer 
system  that  resulted  from  the  process  uses  a  circular  motif  both  for  the  user  interface  and  the 
input  device.  The  input  device  is  a  dial  and  the  user  interface  is  visually  organized  around  the 
concept  of  a  circle.  The  design  process  itself  proceeded  in  the  presence  of  a  great  many 
constraints  and  we  discuss  these  constraints  and  how  an  innovative  design  was  achieved  in  spite 
of  the  constraints. 

[91]  Bans,  J.,  Kruger,  A.,  and  Wahlster,  W.  (January  13,  2002).  “A  Resource-Adaptive  Mobile 
Navigation  System.”  ACM,  lUl  ’02,  pgs.  13-22. 

The  design  of  mobile  navigation  systems  adapting  to  limited  resources  will  be  an  important 
future  challenge.  Since  typically  several  different  means  of  transportation  have  to  be  combined  in 
order  to  reach  a  destination,  the  user  interface  of  such  a  system  has  to  adapt  to  the  user’s 
changing  situation.  This  applies  especially  to  the  alternating  use  of  different  technologies  to 
detect  the  user’s  position,  which  should  be  as  seamless  as  possible.  This  article  presents  a  hybrid 
navigation  system  that  relies  on  different  technologies  to  determine  the  user’s  location  and  that 
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adapts  the  presentation  of  route  directions  to  the  limited  technical  resources  of  the  output  device 
and  the  limited  cognitive  resources  of  the  user. 


[92]  Belloti,  V.,  M.  Back,  W.K.  Edwards,  R.E.  Grinter,  A.  Henderson,  and  C.  Eopes.  (April  20-25, 

2002).  “Making  Sense  of  Sensing  Systems:  Eive  Questions  for  Designers  and  Researchers.” 
Presented  at  CHI  2002,  Minneapolis,  MN,  415-422. 

This  paper  borrows  ideas  from  social  science  to  inform  the  design  of  novel  “sensing”  user- 
interfaces  for  computing  technology.  Specifically,  the  authors  present  five  design  challenges 
inspired  by  analysis  of  human- human  communication  that  are  mundanely  addressed  by 
traditional  graphical  user  interface  designs  (GUIs).  Although  classic  GUI  conventions  allow  us 
to  finesse  these  questions,  recent  research  into  innovative  interaction  techniques  such  as 
‘Ubiquitous  Computing’  and  ‘Tangible  Interfaces’  has  begun  to  expose  the  interaction 
challenges  and  problems  they  pose.  By  making  them  explicit  the  authors  open  a  discourse  on 
how  an  approach  similar  to  that  used  by  social  scientists  in  studying  human-human  interaction 
might  inform  the  design  of  novel  interaction  mechanisms  that  can  be  used  to  handle  human- 
computer  communication  accomplishments. 

[93]  Buyukkokten,  O.,  Kaljuvee,  O.,  Garcia-Molina,  H.,  Paepcke,  A.,  and  Winograd,  T.  (January 
2002).  “Efficient  Web  Browsing  on  Handheld  Devices  Using  Page  and  Eorm  Summarization.” 
ACM  Transactions  on  Information  Systems  20:  82-115. 

The  authors  present  a  design  and  implementation  for  displaying  and  manipulating  HTME  pages 
on  small  handheld  devices  such  as  personal  digital  assistants  (PDAs),  or  cellular  phones.  The 
authors  introduce  methods  for  summarizing  parts  of  Web  pages  and  HTME  forms.  Each  Web 
page  is  broken  into  text  units  that  can  each  be  hidden,  partially  displayed,  made  fully  visible,  or 
summarized.  A  variety  of  methods  are  introduced  that  summarize  the  text  units.  In  addition, 
HTME  forms  are  also  summarized  by  displaying  just  the  text  labels  that  prompt  the  use  for  input. 
The  authors  tested  the  relative  performance  of  the  summarization  methods  by  asking  human 
subjects  to  accomplish  single-page  information  search  tasks.  The  authors  found  that  the 
combination  of  keywords  and  single- sentence  summaries  provides  significant  improvements  in 
access  times  and  number  of  required  pen  actions,  as  compared  to  other  schemes.  The 
experiments  also  show  that  their  algorithms  can  identify  the  appropriate  labels  for  forms  in  95% 
of  the  cases,  allowing  effective  form  support  for  small  screens. 

[94]  Camarata,  K.,  Do,  E.  Y.-E.,  Johnson,  B.R.,  and  Gross,  M.D.  (January  13,  2002).  “Navigational 

Blocks  -  Navigating  Information  Space  with  Tangible  Media.”  ACM,  lUl  ’02,  31-38. 

The  Navigational  Blocks  project  demonstrates  a  tangible  user  interface  that  facilitates  retrieval  of 
historical  stories  in  a  tourist  spot.  Orientation,  movement,  and  relative  positions  of  physical 
Blocks  support  visitor  navigation  and  exploration  in  a  virtual  gallery.  The  Navigational  Blocks 
systems  provides  a  physical  embodiment  of  digital  information  through  tactile  manipulation  and 
haptic  feedback.  The  simple  cubic  form  of  the  Blocks  is  easy  to  understand  and  therefore  easy  to 
use  to  manipulate  complex  digital  information.  Electromagnets  embedded  in  the  Blocks  and 
wireless  communication  encourages  users  to  quickly  rearrange  the  Blocks  to  form  different 
database  queries. 
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[95]  Carpineto,  C.,  G.  Romano,  and  V.  Giannini.  (July  2002).  “Improving  Retrieval  Feedback  with 

Multiple  Term-Ranking  Function  Combination.”  ACM  Transactions  on  Information  Systems  20: 
259-290. 

In  this  article  the  authors  consider  methods  for  automatic  query  expansion  from  top  retrieved 
documents  (i.e.,  retrieval  feedback)  that  make  use  of  various  functions  for  scoring  expansion 
terms  within  Rocchio’s  classical  reweighting  scheme.  An  analytical  comparison  shows  that  the 
retrieval  performance  of  methods  based  on  distinct  term-scoring  functions  is  comparable  on  the 
whole  query  set  but  differs  considerably  on  single  queries,  consistent  with  the  fact  that  the 
ordered  sets  of  expansion  terms  suggested  for  each  query  by  the  different  functions  are  largely 
uncorrelated.  Motivated  by  these  findings,  the  authors  argue  that  the  results  of  multiple  functions 
can  be  merged,  by  analogy  with  ensembling  classifiers,  and  present  a  simple  combination 
technique  based  on  the  rank  values  of  the  suggested  terms.  The  combined  retrieval  feedback 
method  is  effective  not  only  with  respect  to  unexpanded  queries  but  also  to  any  individual 
method,  with  notable  improvements  on  the  system’s  precision.  Furthermore,  the  combined 
method  is  robust  with  respect  to  variation  of  experimental  parameters  and  it  is  beneficial  even 
when  the  same  information  needs  are  expressed  with  shorter  queries. 

[96]  Chen,  X.,  and  Wahls,  T.  (2001).  “A  Graphical  User  Interface  for  Executing  Formal 

Specifications.”  ACM,  SAC,  pgs.  648-652. 

In  software  development,  specification  errors  can  cause  budget  overruns  or  complete  failure  of 
development  projects.  In  an  effort  to  reduce  such  risk  in  software  development,  formal  and 
model-based  specification  languages  have  been  developed  and  applied.  Once  such  specification 
language  is  SPECS-C-i-i-,  which  is  used  to  specify  C-i-i-  classes.  A  large  subset  of  SPECS-C-i-i- 
can  be  executed  by  translating  specifications  to  concurrent  constraint  programs.  In  this  work,  the 
authors  describe  a  graphical  user  interface  that  controls  both  the  translation  and  the  execution  of 
the  resulting  constraint  program.  This  graphical  user  interface  provides  a  convenient  tool  for 
constructing  test  cases,  modifying  specifications,  and  validating  original  or  modified 
specifications.  The  interface  can  be  extended  to  accommodate  other  specification  languages,  as 
well  as  more  complex  test  cases  and  specifications. 

[97]  Chittaro,  E.,  and  Scagnetto,  I.  (November  15,  2001).  “Is  Semitransparency  Useful  for  Navigating 

Virtual  Environments?”  ACM,  VRST  ’01,  pgs.  159-166. 

A  relevant  issue  for  any  Virtual  Environment  (VE)  is  the  navigational  support  provided  to  users 
who  are  exploring  it.  Semitransparency  is  sometimes  exploited  as  a  means  to  see  through 
occluding  surfaces  with  the  aim  of  improving  user  navigation  abilities  and  awareness  of  the  VE 
structure.  Designers  who  make  this  choice  assume  that  it  is  useful,  especially  in  the  case  of  VEs 
with  many  levels  of  occluding  surfaces,  e.g.,  virtual  buildings  or  cities.  This  paper  is  devoted  to 
investigate  this  assumption  with  a  proper  experimental  evaluation  on  users.  Eirst,  the  authors 
discuss  possible  ways  for  improving  navigation,  and  focus  on  implementation  choices  for 
semitransparency  as  a  navigation  aid.  Then,  the  authors  present  and  discuss  the  experimental 
evaluation  they  carried  out.  The  author’s  compared  subjects’  performance  in  three  conditions: 
local  exploitation  of  semitransparency  inside  the  VE,  a  more  global  exploitation  provided  by  a 
bird’s-eye- view,  and  a  control  condition  where  neither  of  the  two  features  was  available. 
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[98]  Cooper,  A.  (1995).  About  Face:  The  Essentials  of  User  Interface  Design,  IDG  Books. 

The  “father”  of  Visual  Basic,  Alan  Cooper,  presents  a  methodology  of  user  interface  design  that 
he  has  distilled  from  many  years  of  creating  award-winning  personal  computer  software.  This 
book  does  not  focus  on  code;  instead  it  discusses  highly  technical  topics  in  clear  English. 
Readers  may  not  agree  with  everything  Cooper  has  to  say  about  software  design,  but  they  will 
find  his  ideas  pertinent,  thought-provoking,  and  perceptive. 

[99]  da  Silva,  P.P.  (2001).  “User  Interface  Declarative  Models  and  Development  Environments:  A 

Survey.”  DSV-IS,  Springer- Verlag,  Berlin,  pgs.  207-226. 

Model-Based  User  Interface  Development  Environments  (MB-UIDEs)  provide  a  context  without 
which  user  interface  declarative  models  can  be  constructed  and  related,  as  part  of  the  user 
interface  design  process.  This  paper  provides  a  review  of  MB-UIDE  technologies  and  a 
framework  for  describing  the  elements  of  a  MB-UIDE.  A  representative  collection  of  14  MB- 
UIDEs  are  selected,  described  in  terms  of  the  framework,  compared  and  analyzed  from  the 
information  available  in  the  literature.  The  framework  can  be  used  as  an  introduction  to  the  MB- 
UIDE  technology  since  it  relates  and  provides  a  description  for  the  terms  used  in  MB-UIDE 
papers.  There  are  three  major  advantages  that  derive  from  the  declarative  user  interface  models 
(UIMs):  (1)  They  can  provide  a  more  abstract  description  of  the  UI  than  UI  descriptions 
provided  by  the  other  UI  development  tools;  (2)  They  facilitate  the  creation  of  methods  to  design 
and  implement  the  UI  in  a  systematic  way  since  they  offer  capabilities:  (a)  to  model  user 
interfaces  using  different  levels  of  abstraction;  (b)  to  incrementally  refine  the  models;  and  (3)  to 
re-use  UI  specifications;  and  (3)  They  provide  the  infrastructure  required  to  automate  tasks 
related  to  the  UI  design  and  implementation  processes.  A  major  disadvantage  of  UIMs  is  the 
complexity  of  the  models  and  their  notations,  which  are  often  hard  to  learn  and  use.  However,  it 
is  expected  that  an  appropriate  environment  should  help  to  overcome  the  UIM’s  complexity, 
providing  features  such  as  graphical  editors,  assistants  and  design  critics  to  support  UI  designers. 
The  development  of  model-based  user  interface  development  environments  (MB-UIDEs)  is  still 
challenging  since  some  essential  problems  related  to  this  technology  are  not  completely  solved. 

[100]  Dumais,  S.,  M.  Banko,  E.  Brill,  J.  Ein,  and  A.  Ng.  (August  11-15-  2002).  “Web  Question 
Answering:  Is  More  Always  Better?”  SIGIR  ’02,  ACM:  291-298. 

This  paper  describes  a  question  answering  system  that  is  designed  to  capitalize  on  the 
tremendous  amount  of  data  that  is  now  available  online.  Most  question  answering  systems  use  a 
wide  variety  of  linguistic  resources.  We  focus  instead  on  the  redundancy  available  in  large 
corpora  as  an  important  resource.  The  authors  use  this  redundancy  to  simplify  the  query  rewrites 
that  are  needed  to  use,  and  to  support  answer  mining  from  returned  snippets.  This  system 
performs  quite  well  given  the  simplicity  of  the  techniques  being  utilized.  Experimental  results 
show  that  question  answering  accuracy  can  be  greatly  improved  by  analyzing  more  and  more 
matching  passages.  Simple  passage  ranking  and  n-gram  extraction  techniques  work  well  in  this 
system  making  it  efficient  to  use  with  many  backend  retrieval  engines. 


[101]  Dumais,  S.  T.  and  Schmitt,  D.G.  (1991).  “Iterative  Searching  in  an  Online  Database.”  Human 
Eactors  Society:  398-402. 

An  experiment  examined  how  people  use  an  online  retrieval  system.  Subjects  solved  general 
topical  search  problems  using  a  database  containing  the  full  text  of  news  articles  (e.g.  find 
articles  about  the  “Background  of  the  new  prime  minister  of  Great  Britain”).  Time,  accuracy  and 
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content  of  the  searches  were  recorded.  Of  particular  interest  was  one  of  two  iterative  search 
methods  available  in  the  interface  -  a  Lookup  function  that  allowed  users  to  explicitly  specify  an 
alternative  query;  and  a  LikeThese  function  that  could  be  used  to  automatically  generate  a  new 
query  using  articles  the  user  marked  as  relevant.  Results  showed  that  subjects  could  easily  use 
both  query  reformulation  methods.  Subjects  generated  much  more  effective  LikeThese  searches 
than  Lookup  searches.  An  analysis  of  individual  subject  differences  suggests  that  the  LikeThese 
method  is  more  accessible  to  a  wide  range  of  users. 

[102]  Fabre,  J.,  Howard,  S.,  and  Smith,  S.  (February  2001).  “Designing  the  Temporal  Aspects  of  the 
User  Interface:  View  from  the  Coal  Face.”  Journal  of  Research  and  Practice  in  Information 
Technology,  33:  54-67. 

The  authors  aim  to  support  user  interface  designers  in  capturing,  representing  and  reasoning 
about  temporal  information.  They  have  developed  a  method  to  support  user  interface  designers  in 
considering  how  the  temporal  aspects  of  software  impact  the  user.  Importantly  the  method  is 
based  on  a  detailed  analysis  of  data  from  a  set  of  situated  interviews  that  capture  the  views  of 
practicing  user  interface  designers.  The  paper  discusses  the  background  research  and  motivation 
for  the  method.  The  evidence  from  this  study  suggests  that  temporal  aspects  of  usability  are 
problematic  in  design  partly  because  they  are  neither  given  properties  of  the  user,  nor  the 
interactive  system.  The  perception  of  time  then  happens  at  a  higher  level  than  the  physical,  and  is 
almost  certainly  an  emergent  feature  of  the  ‘human  in  the  world’.  The  complementarity  across 
interviewees  could  be  enhanced  if  designers,  regardless  of  discipline,  could  articulate  use  of 
TAU  specific  knowledge  and  methods  for  use  when  designing.  This  research  is  intended  to  help 
designers  incorporate  the  four  elements  of  TAU  -  Task  Duration,  Task  Location,  Task 
Contingency,  and  Task  Location  into  a  method,  such  that,  the  emerging  user  experience  is 
consistent  with  the  intended  design  goals.  The  process  needs  to  allow  designers  to  make  explicit 
choices  about  temporal  aspects  of  interaction  and  the  context  in  which  their  designs  are  intended 
to  be  used. 

[103]  Frank,  M.,  Muslea,  M.,  Oh,  J.,  Minton,  S.,  and  Knoblock,  C.  (January  14,  2001).  “An  Intelligent 
User  Interface  for  Mixed-Initiative  Multi-Source  Travel  Planning.”  ACM,  lUI  ’01,  pgs.  85-86. 

A  mixed-initiative  planner  in  the  author’s  context  is  one  in  which  either  the  human  or  the 
computer  can  spontaneously  provide  the  content  of  the  same  input  fields.  A  multi-source  planner 
is  one  that  accesses  multiple  external  information  sources  in  parallel,  using  separate  threads.  This 
type  of  highly  dynamic  user  interface  is  desirable  but  presents  a  challenge  in  “keeping  the  user  in 
control”  because  it  can  be  confusing  to  understand  which  fields  of  the  form  currently  “belong”  to 
the  user,  which  ones  “belong”  to  the  system,  how  these  two  interact,  and  when  and  how  their 
ownership  changes. 

[104]  Gee,  K.  (February  2001)  “The  Ergonomics  of  Hypertext  Narrative:  Usability  Testing  as  a  Tool 
for  Evaluation  and  Redesign.”  ACM  Journal  of  Computer  Documentation  25:  3-16. 

While  usability  research  concentrates  on  evaluating  informational  documents  and  Web  sites, 
significant  insights  can  be  gained  from  performing  usability  testing  on  texts  designed  for 
pleasure  reading,  such  as  hypertext  narratives.  This  article  describes  the  results  of  such  a  test. 
The  results  demonstrate  that  the  navigation  systems  required  for  such  texts  can  significantly 
interfere  with  readers’  ability  to  derive  value  or  pleasure  from  the  fiction.  The  results  emphasize 
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the  importance  of  hypertext  authors  providing  more  linear  paths  through  texts  and  of  simplifying 
the  navigational  apparatus  required  to  read  them. 


[105]  Hackos,  J.  T.  and  Redish,  J.C.  (1998).  User  and  Task  Analysis  for  Interface  Design.  New  York, 
NY.  Wiley  Computer  Publishing. 

User  and  Task  Analysis  for  Interface  Design  helps  you  design  a  great  user  interface  by  focusing 
on  the  most  important  step  in  the  process  -  the  first  one.  You  learn  to  go  out  and  observe  your 
users  at  work,  whether  they  are  employees  of  your  company  or  people  in  customer  organizations. 
You  learn  to  find  out  what  your  users  really  need,  not  by  asking  them  what  they  want,  but  by 
going  through  a  process  of  understanding  what  they  are  trying  to  accomplish.  This  book  includes 
many  examples  of  design  successes  and  challenges  for  products  of  every  kind. 

[106]  Haik,  E.,  Barker,  T.,  Sapsford,  J.,  and  Trainis,  S.  (February  24,  2002).  “Investigation  into 
Effective  Navigation  in  Desktop  Virtual  Interfaces.”  ACM,  Web  3D  ’02,  pgs.  59-66. 

The  study  reported  in  this  paper  is  concerned  with  understanding  and  providing  solutions  to 
commonly  reported  navigation  and  orientation  issues  in  virtual  3D  desktop  environments.  An 
experiment  was  performed  to  investigate  some  of  these  issues.  Twelve  subjects  performed 
simple  tasks  within  a  virtual  desktop  environment  using  navigational  tools  developed  specifically 
for  the  research.  Tasks  were  based  on  locating  information  in  simple  environments.  Time  to 
complete  tasks,  ability  to  remember  the  locations  of  information  within  the  environment  and  user 
attitude  were  measured  and  recorded,  along  with  several  other  important  variables.  Performance 
on  tasks  and  attitude  to  the  environments  was  found  in  many  cases  to  be  related  to  the 
navigational  tools  used  within  the  environments.  The  results  demonstrated  that  constraining  and 
guiding  the  navigation  produces  more  effective  navigation  and  orientation.  It  was  found  that  the 
use  of  a  3D  map  to  assist  navigation  was  most  effective  in  terms  of  time  to  perform  tasks 
(p<0.01).  The  development  of  the  navigational  tools,  the  method  used  in  the  experiment  and  the 
results  of  the  experiment  are  reported  in  this  paper.  The  paper  concludes  with  a  discussion  on 
how  the  findings  of  this  research  relate  to  the  solution  to  general  navigational  and  orientational 
issues  in  3D  virtual  environments. 

[107]  Hollerer,  T.,  Feiner,  S.,  Hallaway,  D.,  Bell,  B.,  Eanzagorta,  M.,  Brown,  D.,  Julier,  S.,  Baillot,  Y., 
and  Rosenblum,  E.  (2001).  “User  Interface  Management  Techniques  for  Collaborative  Mobile 
Augmented  Reality.”  Computers  &  Graphics,  25:  799-810. 

Mobile  augmented  reality  systems  (MARS)  have  the  potential  to  revolutionize  the  way  in  which 
information  is  provided  to  users.  Virtual  information  can  be  directly  integrated  with  the  real 
world  surrounding  the  mobile  user,  who  can  interact  with  it  to  display  related  information,  to 
pose  and  resolve  queries,  and  to  collaborate  with  other  users.  However,  the  authors  believe  that 
the  benefits  of  MARS  will  only  be  achieved  if  the  user  interface  (UI)  is  actively  managed  so  as 
to  maximize  the  relevance  and  minimize  the  confusion  of  the  virtual  material  relative  to  the  real 
world.  This  article  addresses  some  of  the  steps  involved  in  this  process,  focusing  on  the  design 
and  layout  of  the  mobile  user’s  overlaid  virtual  environment.  The  augmented  view  of  the  user’s 
surroundings  presents  an  interface  to  context-dependent  operations,  many  of  which  are  related  to 
the  objects  in  view  -  the  augmented  world  is  the  user  interface.  The  authors  present  three  UI 
design  techniques  that  are  intended  to  make  this  interface  as  obvious  and  clear  to  the  user  as 
possible:  information  filtering,  UI  component  design,  and  view  management.  Information 
filtering  helps  select  the  most  relevant  information  to  present  to  the  user.  UI  component  design 
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determines  the  format  in  which  this  information  should  be  conveyed,  based  on  the  available 
display  resources  and  tracking  accuracy.  For  example,  the  absence  of  high  accuracy  position 
tracking  would  favor  body-  or  screen-stabilized  components  over  world- stabilized  ones  that 
would  need  to  be  exactly  registered  with  the  physical  objects  to  which  they  refer.  View 
management  attempts  to  ensure  that  the  virtual  objects  that  are  displayed  visually  are  arranged 
appropriately  with  regard  to  their  projections  on  the  view  plane.  For  example,  the  relationships 
among  objects  should  be  an  unambiguous  as  possible,  and  physical  and  virtual  objects  should  not 
obstruct  the  user’s  view  of  more  important  physical  or  virtual  objects  in  the  scene.  The  authors 
illustrate  these  interface  design  techniques  using  our  prototype  collaborative,  cross-site  MARS 
environment,  which  is  composed  of  mobile  and  non-mobile  augmented  reality  and  virtual  reality 
systems. 

[108]  Jones,  M.  K.  (1989).  Human-Computer  Interaction:  A  Design  Guide.  Educational  Technology 
Publications,  Englewood  Cliffs,  NJ.  ISBN  0-87778-207-5. 

This  book  provides  guidelines  for  the  design  of  effective  computer-user  interfaces,  intended 
primarily  for  teams  designing  instructional  software  and  related  uses.  Examples  are  provided  of 
how  the  computer  screen  should  be  organized. 

[109]  Kan,  M.-Y.,  and  J.E.  Klavans.  (July  13-17,  2002).  “Using  Eibrarian  Techniques  in  Automatic 
Test  Summarization  for  Information  Retrieval.”  JCDL  ’02,  ACM:  pgs  36-45. 

A  current  application  of  automatic  text  summarization  is  to  provide  an  overview  of  relevant 
documents  coming  from  an  information  retrieval  (IR)  system.  This  paper  examines  how 
Centrifuser,  one  such  summarization  system,  was  designed  with  respect  to  methods  used  in  the 
library  community.  The  authors  have  reviewed  these  librarian  expert  techniques  to  assist 
information  seekers  and  codified  them  into  eight  distinct  strategies.  The  authors  detail  how  they 
have  operationalized  six  of  these  strategies  in  Centrifuser  by  computing  an  information  extract, 
indicative  differences  between  documents,  as  well  as  navigational  links  to  narrow  or  broaden  a 
user’s  query.  An  initial  evaluation  of  the  system  indicates  several  results  are  promising. 
Centrifuser’ s  post-processing  yields  an  improvement  over  the  baseline  user  interface  of 
presenting  output  in  the  standard  IR  framework  user  interface  (as  a  ranked  list  of  relevant 
documents).  Results  hint  that  fully  automatic  post-processed  output  approaches  the  same  level  of 
satisfaction  for  systems  that  require  human  intervention.  Text  is  a  powerful  output  modality  since 
it  is  easy  to  understand  and  is  also  a  prerequisite  for  advanced  user  interfaces  involving  speech. 
Centrifuser’ s  ATS  framework  is  based  on  topic  structure,  and  allows  results  to  be  reported  at 
different  levels  of  granularity,  resulting  in  longer  or  short  summaries.  This  flexibility  is  being 
utilized  in  the  author’s  current  user  interface  project  that  allows  the  summaries  to  be  tailored 
lengthwise  for  different  screen  sizes  (desktop  screens,  PDAs,  and  cell  phone  displays)  and  allows 
contextual  zooming  on  specific  subtopics. 

[110]  Eanday,  I. A.,  and  Myers,  B.A.  (March  2001).  “Sketching  Interfaces:  Toward  More  Human 
Interface  Design.”  Computer,  pgs.  56-64. 

An  interactive  user-interface  design  tool  supports  electronic  sketching,  giving  designers  more 
freedom  to  change  sketches  and  more  flexibility  in  creating  and  evaluating  a  design  prototype. 
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[111]  Landay,  J.  A.  and  Myers,  B.A.  (1995).  Interactive  Sketching  for  the  Early  Stages  of  User 
Interface  Design.  ACM  CHI  ‘95  Conference  Proceedings,  Denver,  Colorado,  pp.  43-50. 

Current  interactive  user  interface  construction  tools  are  often  more  of  a  hindrance  than  a  benefit 
during  the  early  stages  of  user  interface  design.  These  tools  take  too  much  time  to  use  and  force 
designers  to  specify  more  of  the  design  details  than  they  wish  at  this  early  stage.  Most  interface 
designers,  especially  those  who  have  a  background  in  graphic  design,  prefer  to  sketch  early 
interface  ideas  on  paper  or  on  a  whiteboard.  We  are  developing  an  interactive  tool  called  SILK 
that  allows  designers  to  quickly  sketch  an  interface  using  an  electronic  pad  and  stylus.  SILK 
preserves  the  important  properties  of  pencil  and  paper:  a  rough  drawing  can  be  produced  very 
quickly  and  the  medium  is  very  flexible.  However,  unlike  a  paper  sketch,  this  electronic  sketch  is 
interactive  and  can  easily  be  modified.  In  addition,  our  system  allows  designers  to  examine, 
annotate,  and  edit  a  complete  history  of  the  design.  When  the  designer  is  satisfied  with  this  early 
prototype,  SILK  can  transform  the  sketch  into  a  complete,  operational  interface  in  a  specified 
look-and-feel.  This  transformation  is  guided  by  the  designer.  By  supporting  the  early  phases  of 
the  interface  design  life  cycle,  our  tool  should  both  ease  the  development  of  user  interface 
prototypes  and  reduce  the  time  needed  to  create  a  final  interface.  This  paper  describes  our 
prototype  and  provides  design  ideas  for  a  production-level  system. 

[112]  Lauesen,  S.,  and  Haming,  M.B.  (July/ August  2001).  “Virtual  Windows:  Linking  User  Tasks, 
Data  Models,  and  Interface  Design.”  IEEE  Software,  pgs.  67-75. 

User  interface  design  comprises  three  major  activities:  organizing  data  into  a  set  of  windows  or 
frames,  defining  functions  that  let  the  user  control  the  system,  and  designing  the  graphical 
appearance  of  windows  and  functions.  These  design  activities  can  build  on  analysis  results  such 
as  task  analysis  and  data  modeling,  and  they  can  include  checking  activities  such  as  reviews  and 
usability  tests.  The  goal  is  to  create  a  system  that  is  easy  to  learn,  is  easy  to  understand,  and 
supports  user  tasks  efficiently.  The  virtual-windows  technique  is  an  approach  based  on 
employing  data  and  tasks  at  the  same  time.  Part  of  the  approach  is  to  design  and  test  the 
graphical  appearance  before  the  functions  are  defined.  A  virtual  window  is  a  picture  on  an 
idealized  screen.  Compared  to  user  interfaces  designed  traditionally,  the  ones  based  on  virtual 
windows  appear  to  have  several  advantages:  (1)  there  are  fewer  windows;  (2)  there  is  efficient 
task  support  (also  for  task  variants);  (3)  users  can  validate  the  database;  and  (4)  users  better 
understand  the  final  system. 


[113]  Marcus,  A.  Eerrante,  J.V.  Kinnunen,  T.  Kuutti,  K.  and  Sparre,  E.  (1998).  Baby  Eaces:  User- 
Interface  Design  for  Small  Displays.  ACM  CHI  ’98,  Proceedings  of  Human  Eactors  in 
Computing  Systems  Summary,  pgs.  96-97. 

User  interface  conferences  and  literature  dwell  upon  projects  in  which  large  color  palettes,  high 
spatial  resolution,  and  large-size  displays  are  presumed  to  be  available.  Many  consumer 
information  appliances  and  hand-held  devices  are  often  more  limited  in  their  characteristics,  yet 
are  increasingly  important  to  bring  the  power  of  the  Web,  productivity  tools,  databases, 
transactions,  and  entertainment  to  more  and  more  people  on  the  go,  or  at  least  not  in  front  of 
higher-performance  PCs,  NCs,  or  workstations.  Professionals  with  insight  into  the  challenges 
and  achievements  of  designing  graphical  user  interfaces  for  small  displays  will  debate  the  best 
way  to  design  for  products  in  which  many  characteristics  are  significantly  limited,  e.g.  fonts, 
color  resolution,  spatial  resolution,  and  graphics.  These  user  interfaces,  which  might  be  called 
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“baby  faces”  seem  simpler  in  some  ways,  but  also  are  actually  quite  complex  as  a  design 
challenge  when  one  designs  to  account  for  their  limited  characteristics. 


[114]  Mason,  L.  D.  (1997).  “Design  Issues  for  Producing  Effective  Multimedia  Presentations.” 
Technical  Communication,  First  Quarter:  65-71. 

When  designing  multimedia  presentations,  technical  communicators  must  consider  navigational 
aids  and  the  degree  of  user  control,  audio  cues,  color  and  typographical  elements,  visual 
elements,  and  copyright  issues.  Understanding  these  issues  will  help  us  develop  guidelines  for 
effective  use  of  multimedia. 

[115]  McDonald,  S.,  Lai,  T-S.,  and  Tait,  J.  (September  9,  2001)  “Evaluating  a  Content  Based  Image 
Retrieval  System.”  ACM.  SIGIR  ’01,  pgs.  232-240. 

Content  Based  Image  Retrieval  (CBIR)  presents  special  challenges  in  terms  of  how  image  data  is 
indexed,  accessed,  and  how  end  systems  are  evaluated.  This  paper  discusses  the  design  of  a 
CBIR  system  that  uses  global  colour  as  the  primary  indexing  key,  and  a  user  centered  evaluation 
of  the  system’s  visual  search  tools.  The  results  indicate  that  users  are  able  to  make  use  of  a  range 
of  visual  search  tools,  and  that  different  tools  are  used  at  different  points  in  the  search  process. 
The  results  also  show  that  the  provision  of  a  structured  navigation  and  browsing  tool  can  support 
image  retrieval,  particularly  in  situations  in  which  the  user  does  not  have  a  target  image  in  mind. 
The  results  are  discussed  in  terms  of  their  implications  for  the  design  of  visual  search  tools,  and 
their  implications  for  the  use  of  user-centered  evaluation  for  CBIR  systems. 


[1 16]  Mehlenbacher,  B.  (1992).  Navigating  Online  Information:  A  Characterization  of  Extralinguistic 
Factors  That  Influence  User  Behavior.  ACM  SIGDOC  ’92,  Proceedings  of  the  10*  Annual 
International  Conference  on  Systems  Documentation,  Ottawa,  Canada. 

This  paper  examines  the  extralinguistic  factors  that  influence  user  behavior  with  online 
information  systems.  Extralinguistic  factors  include  any  interface  features  that  are  “outside”  how 
users  understand  and  comprehend  written  texts  online.  Extralinguistic  features,  therefore,  are 
interface  features  that  support  (1)  how  users  formulate  their  information  goals  or  represent  their 
tasks,  (2)  how  users  navigate  to  new  or  related  topics  of  interest  to  them,  and  (3)  how  users 
quickly  scan  (rather  than  read)  online  information.  It  is  argued  that  text  comprehension  is  only 
one  task  that  users  engage  in  when  using  online  information  systems.  A  model  of  online  user 
behavior  that  includes  goal  setting,  navigating,  scanning,  and  text  comprehension  is  outlined. 
The  author  argues  that  a  broader  definition  of  online  information  use  is  necessary  and  discusses 
various  design  principles  for  avoiding  communication  breakdowns  before  users  reach  their 
desired  information.  Finally,  the  author  concludes  by  suggesting  that  a  Participatory  Design 
Approach  to  the  design  of  human-computer  interfaces  is  one  method  of  undermining  our 
tendency,  as  software  designers,  to  apply  design  advice  and  guidelines  without  first  accounting 
for  user  tasks  and  information  goals. 


[117]  Mullet,  K.  and  Sano,  D.  (1995).  Designing  User  Interfaces,  SunSoft  Press. 

This  book  describes  fundamental  techniques  that  can  be  used  to  enhance  the  visual  quality  of 
graphical  user  interfaces,  data  displays,  and  multimedia  productions.  The  authors  approach 
interface  design  from  the  perspective  of  communication-oriented  graphic  design,  industrial 
design,  and  architecture.  The  skills,  practice,  and  orientation  of  these  design  disciplines  are 
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essential  to  the  success  of  every  graphical  interface,  yet  few  software  developers,  human  factors 
professionals,  or  product  managers  receive  any  training  in  these  areas.  This  book  addresses  this 
shortcoming  by  describing  essential  design  concepts  as  manifested  in  the  graphical  user 
interface. 

[118]  Riedl,  M.O.,  and  Amant,  R.S.  (January  13,  2002).  “Toward  Automated  Exploration  of 
Interactive  Systems.”  ACM,  lUl  ’02,pgs.  135-142. 

The  ease  with  which  a  user  interface  can  be  navigated  strongly  contributes  to  its  usability.  In  this 
paper  the  authors  describe  preliminary  results  of  a  project  aimed  at  making  the  evaluation  of  user 
interfaces  form  this  perspective  more  routine.  The  authors  have  designed  a  system  to  carry  out  an 
autonomous,  exploratory  navigation  through  the  graphical  user  interface  of  interactive,  off-the- 
shelf  software  applications.  The  system  is  not  a  robust  tool,  but  rather  a  proof  of  concept  that  can 
exhibit  interesting  behaviors.  The  traversal  process  generates  a  representation  of  the  connectivity 
of  the  user  interface,  as  well  as  navigational  paths  to  specific  commands.  The  reasoning 
component  of  the  system  is  based  on  the  ACT-R  architecture,  while  the  perceptual  and  motor 
components  of  the  system  are  built  on  top  of  the  SegMan  perception/action  substrate.  The 
authors  present  the  design  of  the  system  and  its  use  in  exploring  a  simple  user  interface. 


[119]  Shneiderman,  B.  (1998).  Designing  the  User  Interface:  Strategies  for  Effective  Human- 
Computer  Interaction,  Third  Edition.  Addison  Wesley  Longman,  Inc.,  Reading,  MA. 

This  book  is  intended  primarily  for  designers,  managers,  and  evaluators  of  interactive  systems. 
The  author  presents  a  broad  survey  of  designing,  implementing,  managing,  maintaining,  training, 
and  refining  the  user  interface  of  interactive  systems.  The  author  offers  practical  techniques  and 
guidelines  for  interface  design  and  discusses  underlying  issues  and  supports  conclusions  with 
empirical  results.  Coverage  includes  the  human  factors  of  interactive  software  (with  a  discussion 
of  diverse  user  communities),  tested  methods  to  develop  and  assess  interfaces,  interaction  styles 
such  as  direct  manipulation  for  graphical  user  interfaces,  and  design  considerations  such  as 
effective  messages,  consistent  screen  design,  and  appropriate  color. 

[120]  Stoll,  G.,  Eldridge,  M.,  Patterson,  D.,  Webb,  A.,  Berman,  S.,  Levy,  R.,  Caywood,  C.,  Taveira, 

M.,  Hunt,  S.,  and  Hanrahan,  P.  (August  2001).  “Lightning-2:  A  High-Performance  Display 
Subsystem  for  PC  Clusters.”  ACM,  SigGraph  2001,  pgs.  141-148. 

Clusters  of  PCs  are  increasingly  popular  as  cost-effective  platforms  for  supercomputer-class 
applications.  Given  recent  performance  improvements  in  graphics  accelerators,  clusters  are 
similarly  attractive  for  demanding  graphics  applications.  The  authors  describe  the  design  and 
implementation  of  Lightning-2,  a  display  subsystem  for  such  a  cluster.  The  system  scales  in  both 
the  number  of  rendering  nodes  and  the  number  of  displays  supported,  and  allows  any  pixel  data 
generated  from  any  node  to  be  dynamically  mapped  to  any  location  on  any  display.  A  number  of 
image-compositing  functions  are  supported,  including  color-keying  and  depth-compositing.  A 
distinguishing  feature  of  the  system  is  its  platform  independence:  it  connects  to  graphics 
accelerators  via  an  industry- standard  digital  video  port  and  requires  no  modifications  to 
accelerator  hardware  or  device  drivers.  As  a  result,  rendering  clusters  that  utilize  Lightning-2  can 
be  upgraded  across  multiple  generations  of  graphics  accelerators  with  little  effort. 
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[121]  Wu,  H.,  de  Kort,  E.,  and  De  Bra,  P.  (August  2001).  “Design  Issues  for  General-Purpose 
Adaptive  Hypermedia  Systems.”  ACM,  HT  ’01,  pgs.  141-150. 

A  hypermedia  application  offers  its  users  much  freedom  to  navigate  through  a  large  hyperspace. 
For  authors  finding  a  good  compromise  between  offering  navigational  freedom  and  offering 
guidance  is  difficult,  especially  in  applications  that  target  a  broad  audience.  Adaptive 
hypermedia  (AH)  offers  (automatically  generated)  personalized  content  and  navigation  support, 
so  the  choice  between  freedom  and  guidance  can  be  made  on  an  individual  basis.  Many  adaptive 
hypermedia  systems  (AHS)  are  tightly  integrated  with  one  specific  application.  In  this  paper  the 
authors  study  design  issues  for  general-purpose  adaptive  hypermedia  systems,  built  according  to 
an  application-independent  architecture.  They  use  the  Dexter-based  AHAM  reference  model  for 
adaptive  hypermedia  to  describe  the  functionality  of  such  systems  at  the  conceptual  level.  The 
authors  concentrate  on  the  architecture  and  behavior  of  a  general-purpose  adaptive  engine.  Such 
an  engine  performs  adaptation  and  updates  the  user  model  according  to  a  set  of  adaptation  rules 
specified  in  an  adaptation  model.  The  authors  study  the  behavior  of  such  a  system  and 
concentrate  on  the  issues  of  termination  and  confluence,  which  are  important  to  detect  potential 
problems  in  an  adaptive  hypermedia  application.  The  authors  draw  parallels  with  static  rule 
analysis  in  active  database  systems.  By  using  common  properties  of  AHS  we  are  able  to  obtain 
more  precise  (less  conservative)  results  for  AHS  than  for  active  databases  in  general,  especially 
for  the  problem  of  termination. 


h.  Visualization 

[122]  Ames,  A.L.  (October  21,  2001).  “Communicating  Effectively  with  Interaction.”  ACM,  SIGDOC 
W,pgs.  1-6. 

The  ability  to  build  interactions  that  support,  enable,  and  improve  communication  is  a  valuable 
skill  for  help  developers.  Web-site  designers,  multimedia  content  developers,  information-rich 
user  interface  designers  -  anyone  who  designs  and  develops  information  to  be  used  online.  This 
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human  identifies  promising  regions  of  the  search  space  for  the  computer  to  explore,  and  also 
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This  paper  argues  for  the  design  of  collaborative  visual  interfaces  to  digital  libraries  that  support 
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The  visual  image  presented  to  an  engineer  using  a  computer-aided  design  (CAD)  system 
influences  design  activities  such  as  decision  making,  problem  solving,  cognizance  of  complex 
relationships,  and  error  correction.  Because  of  the  three-dimensional  (3-D)  nature  of  the  object 
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spatial  characteristics  in  a  quick  and  direct  manner.  Subjects  were  instructed  to  memorize  the 
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during  the  study  period  and  during  the  trials  that  followed.  Subject  response  time  and  error  rate 
were  evaluated.  Improved  performance  due  to  hidden  surface  is  the  most  convincing 
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experimental  finding.  Interposition  is  a  powerful  cue  to  object  structure  and  should  not  be  limited 
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cognitive  processes  involved  in  viewing  visualizations.  Effective  visualization  systems  support  a 
broad  range  of  user  tasks  and  abilities,  are  easy  to  learn,  and  provide  powerful  and  flexible 
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[131]  Gallimore,  J.  J.  and  Brown,  M.E.  (1993)  “Visualization  of  3-D  Computer-Aided  Design 
Objects.”  International  Journal  of  Human-Computer  Interaction  5(4):  361-382. 

There  are  many  significant  human  factors  issues  associated  with  the  design  and  visualization  of 
complex,  computer- generated  images.  One  such  issue  is  to  determine  the  most  effective 
techniques  for  providing  engineers  with  realistic  three-dimensional  (3-D)  objects.  Differences  in 
subjects’  ability  to  discriminate  between  the  shape  of  two  3-D,  perspective,  computer-aided 
design-type  (CAD-type)  objects  were  investigated  for  various  levels  of  monocular  coding 
techniques  and  the  binocular  cue  stereopsis.  Performance  was  assessed  at  all  combinations  of 
five  levels  of  monocular  cues  and  two  levels  of  disparity.  The  task  was  similar  to  the  classic 
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mental  rotation  paradigm  except  that  subjects  were  provided  with  the  ability  to  rotate  one  of  the 
objects  using  a  two-dimensional  (2-D)  joystick.  Results  indicate  that  interposition  was  the  only 
depth  cue  that  significantly  enhanced  subjects’  performance.  One  explanation  for  the  lack  of 
significance  of  stereopsis  is  that,  for  this  particular  task,  depth  information  provided  by  disparity 
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that  subjects  employed  a  feature-by-feature  comparison  task  strategy.  With  tools  that  can 
evaluate  cognitive  activity  such  as  the  mental  rotation  paradigm  and  orientation  index, 
researchers  can  learn  more  about  how  designers  visualize  and  understand  3-D  CAD  objects. 
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solve  this  problem.  The  authors  introduced  the  concepts  of  legibility  and  cognitive  maps  and  the 
five  features  of  urban  landscape  with  which  they  are  associated.  Eollowing  this  are  descriptions 
of  techniques  and  algorithms  which  we  have  developed  to  allow  these  features  to  be  introduced 
to  three  dimensional  spaces  for  information  visualization.  Then  they  describe  a  specific 
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This  paper  describes  a  novel  method  for  the  visualization  of  hierarchically  structured 
information.  The  Tree-Map  visualization  technique  makes  100%  use  of  the  available  display 
space,  mapping  the  full  hierarchy  onto  a  rectangular  region  in  a  space-filling  manner.  This 
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efficient  use  of  space  allows  very  large  hierarchies  to  be  displayed  in  their  entirety  and  facilitates 
the  presentation  of  semantic  information. 
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This  paper  presents  a  reactive  interface  display  which  allows  information  seekers  to  explore 
complex  information  spaces.  We  have  adopted  information  seeking  dialogue  as  a  fundamental 
model  of  interaction  and  implemented  a  prototype  system  in  the  mapping  domain — GeoSpace — 
which  progressively  provides  information  upon  a  user’s  input  queries.  Domain  knowledge  is 
represented  in  a  form  of  information  presentation  plan  modules,  and  an  activation  spreading 
network  technique  is  used  to  determine  the  relevance  of  information.  The  reactive  nature  of  the 
activation  spreading  network,  combined  with  visual  design  techniques,  such  as  typography,  color 
and  transparency  enables  the  system  to  support  the  information  seeker  in  exploring  the  complex 
information  space.  The  system  also  incorporates  a  simple  learning  mechanism  which  enables  the 
system  to  adapt  the  display  to  a  particular  user’s  preferences.  GeoSpace  allows  users  to  rapidly 
identify  information  in  a  dense  display  and  it  can  guide  a  users’  attention  in  a  fluid  manner  while 
preserving  overall  context. 
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This  paper  has  the  intention  to  present  new  ideas  for  data  mining  and  visualization  of  financial 
information.  On  this  occasion  new  methods  for  the  graphical  representation  of  abstract  and 
multi-dimensional  information  and  information  structures  are  discussed.  This  will  enable  users  to 
get  an  overview  of  the  global  economy  or  an  insight  in  the  development  of  economic  key  data. 
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Visualizing  a  construction  schedule  helps  planners  to  identity  potential  construction  problems 
prior  to  actual  building  construction.  Planners  must  envision  the  sequence  of  construction,  the 
workspace  logistics,  and  utilization  of  resources  and  equipment  in  space  and  over  time.  This 
paper  discusses  methods  of  generating,  visualizing,  and  evaluating  construction  planning 
information  with  CAD  based  tools.  We  use  a  construction  example  to  illustrate  how  feature 
extraction  of  3D  CAD  models  can  help  identify  construction  problems  and  evaluate  the  quality 
of  a  construction  plan  through  4D  analysis  and  4D  annotation. 
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Overview  diagrams  are  one  of  the  best  tools  for  orientation  and  navigation  in  hypermedia 
systems.  However,  constructing  effective  overview  diagrams  is  a  challenging  task.  This  paper 
describes  the  Navigational  View  Builder,  a  tool  which  allows  the  user  to  interactively  create 
useful  visualizations  of  the  information  space.  It  uses  four  strategies  to  form  effective  views. 
These  are  binding,  clustering,  filtering  and  hierarchization.  These  strategies  use  a  combination  of 
structure  and  content  analysis  of  the  underlying  space  for  forming  the  visualizations.  This  paper 
discusses  these  strategies  and  shows  how  they  can  be  applied  for  forming  visualizations  for  the 
World-Wide  Web. 
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This  section  of  the  report  takes  a  look  at  a  few  of  the  information  visualization  systems  available. 
It  is  split  into  two  areas,  the  first  being  2D  systems  and  the  second  being  3D  systems.  To  analyze 
every  single  graphical  information  display  system  would  require  a  book,  so  this  section  takes  a 
look  at  those  methods  which  are  either  new  or  of  direct  relevance  to  this  research. 
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This  paper  presents  a  historical  overview  of  visualization  as  a  cognitive  strategy  in  human 
creativity,  discovery,  and  problem-solving.  Visualization  strategies,  such  as  mental  imagery, 
pervade  historical  accounts  of  scientific  discovery  and  invention.  A  selected  number  of  historical 
examples  are  presented  and  discussed  on  topics  such  as  physics,  aviation,  and  the  science  of 
chaos.  Everyday  examples  are  also  discussed  to  show  the  value  of  visualization  in  cognition  for 
all  people.  Several  counter  examples  are  also  discussed  showing  that  visualization  can  also  lead 
to  erroneous  conclusions.  Many  educational  implications  are  discussed,  such  as  reconsidering  the 
dominant  role  and  value  schools  place  on  verbal,  abstract  thinking.  These  issues  are  also 
considered  in  light  of  emerging  computer-based  technologies,  such  as  virtual  reality. 
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We  consider  designing  as  a  conversation  with  materials  conducted  in  the  medium  of  drawing  and 
crucially  dependent  on  seeing,  of  which  we  distinguish  several  kinds:  literal  visual  apprehension 
of  (in  this  instance)  marks  on  a  page;  appreciative  judgments  of  quality  (such  as  ‘too  small  in 
scale’);  apprehension  of  spatial  gestalts  (such  as  ‘two  L’s  back  to  back’).  We  describe  the 
functions  of  several  kinds  of  seeing  in  designers’  move  experiments,  their  reasoning,  and  the 
learning  that  results  from  and  feeds  into  episodes  of  designing.  From  this  analysis,  we  draw 
implications  for  epistemology,  education  and  computation  in  design. 
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This  book  describes  design  strategies  -  the  proper  arrangement  in  space  and  time  of  images, 
words,  and  numbers  -  for  presenting  information  about  motion,  process,  mechanism,  cause  and 
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effect.  These  strategies  are  found  again  and  again  in  portrayals  of  explanations,  quite 
independent  of  the  particular  substantive  content  or  technology  of  display.  The  first  part  of  the 
book  examines  the  logic  of  depicting  quantitative  evidence.  The  second  part  considers  design 
strategies,  often  for  the  arrangement  of  images  as  narrative. 
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Ease  of  use  is  a  primary  concern  of  serious  developers  faced  with  integrating  a  dynamic  graphics 
software  product  into  interactive  applications.  Specifically,  the  developer  must  understand  how 
easy  it  will  be  to  attain  the  desired  result  while  maintaining  maximum  application  performance. 
This  paper  defines  six  categories  for  consideration  when  evaluating  a  dynamic  graphics 
subsystem  for  ease-of-lee:  optimal  performance,  minimal  programming,  design 
openness/flexibility,  integration  with  native  widgets,  advanced  tools  and  features,  and  portability. 
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The  world  is  complex,  dynamic,  multidimensional;  the  paper  is  static,  flat.  How  are  we  to 
present  the  rich  visual  world  of  experience  and  measurement  on  mere  flatland?  This  book 
celebrates  escapes  from  flatland,  rendering  several  hundred  superb  displays  of  complex  data.  The 
author’s  investigation  yields  general  principles  that  have  specific  visual  consequences,  governing 
the  design,  editing,  analysis,  and  critique  of  data  representations.  These  principles  help  to 
identify  and  to  explain  design  excellence  -  why  some  displays  are  better  than  others.  Charts, 
diagrams,  graphs,  tables,  guides,  instructions,  directories,  and  maps  comprise  an  enormous 
accumulation  of  material.  This  book  arrays  exemplary  designs  -  this  time  over  a  broader 
spectrum,  for  all  types  of  information. 


[146]  Tufte,  E.  (1983).  The  Visual  Display  of  Quantitative  Information.  Graphics  Press:  Cheshire, 
Connecticut. 

Data  graphics  visually  display  measured  quantities  by  means  of  the  combined  use  of  points, 
lines,  a  coordinate  system,  numbers,  symbols,  words,  shading,  and  color.  The  first  part  of  this 
book  reviews  the  graphical  practice  of  the  two  centuries  since  Playfair.  The  second  part  of  the 
book  provides  a  language  for  discussing  graphics  and  a  practical  theory  of  data  graphics. 
Applying  to  most  visual  displays  of  quantitative  information,  the  theory  leads  to  changes  and 
improvements  in  design,  suggests  why  some  graphics  might  be  better  than  others,  and  generates 
new  types  of  graphics.  This  is  a  book  about  the  design  of  statistical  graphics  and,  as  such,  it  is 
concerned  both  with  design  and  with  statistics.  But  it  is  also  about  how  to  communicate 
information  through  the  simultaneous  presentation  of  words,  numbers,  and  pictures. 

[147]  Wagner,  I.,  M.  Kompast,  and  R.  Earner.  (September  /  October  2002).  “Visualization  Strategies 
for  the  Design  of  Interactive  Navigable  3-D  Worlds.”  ACM  Interactions,  pgs  25-34. 

Designing  navigable,  interactive  3-D  worlds  is  partly  an  aesthetic  and  partly  a  technical  question. 
While  having  to  solve  the  technical  problems  of  optimizing  performance  in  practice,  the  authors 
focused  on  the  challenge  of  creating  designs  that  generate  a  different  view  on  things,  helping 
perceive  the  novel  in  the  familiar,  discover  relationships  between  seemingly  incongruous  objects, 
and  relate  the  unrelatable.  Obviously  realism  is  not  an  answer  to  this  challenge.  The  designer 
team  approached  it  through  bringing  different  design  principles  together,  pooling  their 
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knowledges  and  expertise.  The  design  principles  the  author  present  in  this  article  are  a  result  of 
this  work.  At  the  core  of  their  development  was  an  intense  discussion  of  a  great  diversity  of 
visual  examples  of  artwork:  paintings,  film,  stage  design,  comics.  The  designer  team 
experimented  with  different  strategies  of  combing  the  visual  languages  of  these  artworks  into  3- 
D  scenographies.  The  design  principles,  although  described  textually,  have  been  made  visible 
through  particularly  telling  views  of  the  3-D  worlds.  The  authors  look  at  these  screenshots  as 
visual  examples  of  directions  to  take  in  3-D  design  rather  than  as  accomplished  prototypes. 


i.  Other 

[148]  Benoff,  D.  (November  2000).  “Interactive  Electronic  Technical  Manuals.”  Maintenance,  87:  88- 
93. 

Electronic  publications  can  increase  the  efficiency  of  your  digital  aircraft  and  analog  technicians. 
Computerized  technical  manuals  are  silently  revolutionizing  the  aircraft  maintenance  industry  by 
helping  the  technician  isolate  problems  quickly,  and  in  the  process  reduce  downtime  and  costs  by 
more  than  10  percent.  These  electronic  publications  can  reduce  the  numerous  volumes  of 
maintenance  manuals,  microfiche  and  work  cards  that  are  used  to  maintain  engines,  airframes, 
avionics,  and  their  associated  components. 

[149]  Blue,  R.S.,  J.  Wampler,  G.  Bowden  Wise,  E.J.  Hoebel,  B.  Yamrom,  C.R.  Volpe,  B.  Wilde,  P. 
Rondot,  A.E.  Kelly,  A.  Gilman,  W.  Turner,  S.  Einthicum,  G.  Ryon.  (April  20-25,  2002).  “An 
Automated  Approach  and  Virtual  Environment  for  Generating  Maintenance  Instructions.”  CHI 
2002,  ACM,  pgs.  494-495. 

Maintenance  of  complex  machinery  such  as  aircraft  engines  requires  reliable  and  accurate 
documentation,  including  illustrated  parts  catalogs  (IPGs),  exploded  views,  and  technical 
manuals  describing  how  to  remove,  inspect,  repair  and  install  parts.  Eor  new  designs,  there  are 
often  time  constraints  for  getting  a  new  engine  to  the  field,  and  the  available  documentation  must 
go  with  it.  Authoring  technical  manuals  is  a  complex  process  involving  technical  writers, 
engineers,  as  well  as  domain  experts  (mechanics  and  designers).  Often,  several  revisions  are 
required  before  a  manual  has  correct  IPC  figures  and  maintenance  instructions.  Compounding 
this  problem  is  that  technical  writers  often  perform  tasks  better  suited  for  computers,  leading  to 
increased  costs  and  errors.  In  this  demonstration,  the  authors  describe  a  new  framework  to 
generate  maintenance  instructions  from  solid  models  (Computer  Aided  Design/CAD  data)  and 
then  validate  these  instructions  in  a  haptics-enabled  virtual  environment.  The  authors  approach 
utilizes  natural  language  processing  techniques  to  generate  a  presentation-independent  logical 
form,  which  can  be  transformed  for  display  within  the  virtual  environment.  During  the 
development  of  the  system,  task  analyses,  human  models,  usability  studies,  and  domain  experts 
were  used  to  gain  insights.  The  end  result  is  a  more  integrated  and  human-centered  process  for 
developing  technical  manuals,  providing  higher  quality  documents  with  less  cost. 

[150]  Bursky,  D.  (June  18,  2001).  “Single  Instrument  Analyze  Protocols  and  Eogic.”  Electronic 
Design,  pgs.  66-72. 

By  combining  intelligence  and  an  easy-to-leam  user  interface,  the  Bus  Doctor  simplifies  system 
diagnostics.  System  designers  today  must  deal  with  complex  buses  and  equally  complex  bus¬ 
signaling  protocols.  This  often  requires  using  separate  instruments,  each  optimized  for  either 
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logic  analysis  or  protocol  analysis.  But  advances  in  bus  operating  speeds  and  changes  in 
protocols  frequently  cut  the  instrument’s  life  short.  The  ideal  solution  would  combine  the  ability 
to  handle  both  logic  and  protocols,  while  also  allowing  easy  upgrades  to  support  higher  speeds 
and  multiple  protocols.  Designers  at  Data  Transit  have  crafted  exactly  this  combination.  Known 
as  the  Bus  Doctor,  it  performs  both  logic  and  protocol  analysis.  To  do  this,  the  instrument 
leverages  programmable  logic  and  software-based  control  to  allow  instant  reconfiguration  and 
feature  upgrades. 

[151]  Creed,  A.  Dennis,  I.  and  Newstead,  S.  (1987)  “Proof-reading  on  VDUs.”  Behaviour  and 
Information  Technology  6(1):  3-13. 

Two  experiments  are  reported  which  compared  proofreading  performance  across  three  different 
modes  of  presentation.  The  results  of  Experiment  1  indicated  that  proofreading  accuracy  was 
significantly  worse  on  a  VDU  than  on  paper,  with  a  photograph  of  the  VDU  display  producing 
intermediate  performance.  It  was  also  demonstrated  that  substitutions  of  visually  similar  errors 
were  harder  to  detect  than  other  error  types.  The  results  of  Experiment  2  replicated  these 
findings,  but  failed  to  find  any  difference  between  twin-column  and  single-column  presentation. 
The  findings  are  interpreted  as  indicating  that  character  font  may  be  a  major  factor  in  the  poor 
performance  with  the  VDU.  It  is  suggested  that  the  present  experimental  paradigm  could  be  a 
useful  assessment  tool  for  the  evaluation  of  display  design. 

[152]  Cronen-Townsend,  S.,  Y.  Zhou,  and  W.B.  Croft.  (August  11-15,  2002).  “Predicting  Query 
Performance.”  SIGIR  ’02,  ACM:  299-306. 

The  authors  develop  a  method  for  predicting  query  performance  by  computing  the  relative 
entropy  between  a  query  language  model  and  the  corresponding  collection  language  model.  The 
resulting  clarity  score  measures  the  coherence  of  the  language  usage  in  documents  whose 
models  are  likely  to  generate  the  query.  The  authors  suggest  that  clarity  scores  measure  the 
ambiguity  of  a  query  with  respect  to  a  collection  of  documents  and  show  that  they  correlate 
positively  with  average  precision  in  a  variety  of  TREC  test  sets.  Thus,  the  clarity  score  may  be 
used  to  identify  ineffective  queries,  on  average,  without  relevance  information.  The  authors 
develop  an  algorithm  for  automatically  setting  the  clarity  score  threshold  between  predicted 
poorly-performing  queries  and  acceptable  queries  and  validate  it  using  TREC  data.  In  particular, 
the  authors  compare  the  automatic  thresholds  to  optimum  thresholds  and  also  check  how 
frequently  results  as  good  are  achieved  in  sampling  experiments  that  randomly  assign  queries  to 
the  two  classes. 

[153]  Dodson,  D.  Diagrammatic  Interaction  Group  (DIG), 
web.soi.city.ac.uk/research/dig/dig.html 

DIG  conducts  research  into  human-computer  interaction  through  node-and-link  diagrams.  Its 
work  has  mainly  concerned  3D  interactive  diagrams  and  diagrams  that  ‘self-organize’  to 
accommodate  ongoing  change  their  contents.  Its  interests  span  theory,  supporting  technology  and 
applications,  notably  in  user  interfaces  for  knowledge-system  developers  and  end-users. 
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[154]  Gemperle,  F.,  C.  Kasabach,  J.  Stivoric,  M.  Bauer,  and  R.  Martin.  (October  1998).  “Design  for 
Wearability.”  Institute  for  Complex  Engineered  Systems,  Carnegie  Mellon  University.  Second 
International  Symposium  on  Wearable  Computing,  Pittsburgh,  PA. 

Digital  Technology  is  constantly  improving  as  information  becomes  wireless.  These  advances 
demand  more  wearable  and  mobile  form  factors  for  products  that  access  information.  A  product 
that  is  wearable  should  have  wearability.  This  paper  explores  the  concept  of  dynamic 
Wearability  through  design  research.  Wearability  is  defined  as  the  interaction  between  the 
human  body  and  the  wearable  object.  Dynamic  wearability  extends  that  definition  to  include  the 
human  body  in  motion.  This  research  has  been  to  locate,  understand,  and  define  the  spaces  on  the 
human  body  where  solid  and  flexible  forms  can  rest  -  without  interfering  with  fluid  human 
movement.  The  result  is  a  set  of  design  guidelines  embodied  in  a  set  of  wearable  forms.  These 
wearable  forms  describe  the  three  dimensional  spaces  on  the  body  best  suited  for  comfortable 
and  unobtrusive  Wearability  by  design. 

[155]  Gustman,  S.,  D.  Soergel,  D.  Oard,  W.  Byrne,  M.  Picheny,  B.  Ramadhadran,  and  D.  Greenberg. 
(July  13-17,  2002).  “Supporting  Access  to  Large  Digital  Oral  History  Archives.”  JCDL  ’02, 
ACM:  18-22. 

This  paper  describes  the  authors’  experiences  with  the  creation,  indexing,  and  provision  of 
access  to  a  very  large  archive  of  videotaped  oral  histories.  The  research  identifies  a  set  of  critical 
research  issues  that  must  be  addressed  to  be  able  to  provide  full  and  detailed  access  to  collections 
of  this  size:  issues  in  user  requirement  studies,  automatic  speech  recognition,  automatic 
classification,  segmentation,  summarization,  retrieval,  and  user  interfaces. 


[156]  Hackos,  J.  T.  (1994).  “Publications  and  Training:  The  Challenge  of  Working  Together.” 
Technical  Communication  41(3):  424-431. 

When  technical  publications  and  technical  training  departments  learn  to  work  together 
effectively,  everyone  benefits.  Technical  writers  learn  from  trainers  the  details  about  the  users 
and  their  tasks;  technical  trainers  get  help  in  producing  better  training  materials;  users  get 
training  courses  and  technical  manuals  that  are  thoroughly  integrated;  the  company  displays  a 
unified  approach  to  training  and  information. 


[157]  Haydon,  L.  M.  (1997).  The  Complete  Guide  to  Writing  and  Producing  Technical  Manuals.  New 
York,  N.Y.  A  Wiley-Interscience  Publication.  John  Wiley  &  Sons,  Inc. 

This  book  shows  the  reader  how  to  create  clear,  well-organized  technical  manuals  for  any 
equipment,  simple  or  complex.  Requiring  no  specialized  background  knowledge,  this  unique 
guide  lays  out  all  the  aspects  of  the  job  -  from  initial  concept  to  final  publication.  The  author 
draws  on  more  than  twenty-five  years’  experience  as  a  technician  and  technical  writer  to  provide 
authoritative,  easy-to-follow  instructions  on  how  to  organize  detailed  technical  information  into  a 
finished,  high-quality  technical  manual.  Major  topics  include:  planning  procedures  for  technical 
manuals;  manual  types  and  arrangements,  including  operation  manuals,  maintenance  and  repair 
instructions,  illustrated  parts  lists,  and  more;  layout  and  format,  including  sample  page  layouts; 
writing  style  and  technical  editing  techniques;  front  matter  and  introductions;  illustration  and 
table  preparation,  including  typical  charts,  diagrams,  and  illustrations;  preparing  camera-ready 
copy;  printing  and  binding;  organizing  a  technical  handbook  department;  and  much  more. 


©  Copyright  2002,  Carnegie  Mellon  University 
October  2002 


51 


[158]  Hegarty,  M.  (1992).  “Mental  Animation:  Inferring  Motion  from  Static  Diagrams  of  Mechanical 
Systems.”  Journal  of  Experimental  Psychology:  Learning,  Memory  and  Cognition  18(5):  1084- 
1102. 

Reaction-time  and  eye-fixation  data  are  analyzed  to  examine  how  people  infer  the  kinematics  of 
simple  mechanical  systems  from  diagrams  showing  their  static  configuration.  The  mental 
animation  process  reflects  both  capacity  limitations  and  limitations  of  mechanical  knowledge. 

[159]  Hiemstra,  D.  (August  11-15,  2002).  “Term-Specific  Smoothing  for  the  Language  Modeling 
Approach  to  Information  Retrieval:  The  Importance  of  a  Query  Term.”  SIGIR  ’02,  ACM:  35-41. 
This  paper  follows  a  formal  approach  to  information  retrieval  based  on  statistical  language 
models.  By  introducing  some  simple  reformulations  of  the  basic  language  modeling  approach  the 
authors  introduce  the  notion  of  importance  of  a  query  term.  The  importance  of  a  query  term  is  an 
unknown  parameter  that  explicitly  models  which  of  the  query  terms  are  generated  from  the 
relevant  documents  (the  important  terms),  and  which  are  not  (the  unimportant  terms).  The  new 
language  modeling  approach  is  shown  to  explain  a  number  of  practical  facts  of  today’s 
information  retrieval  systems  that  are  not  very  well  explained  by  the  current  state  of  information 
retrieval  theory,  including  stop  words,  mandatory  terms,  coordinator  level  ranking  and  retrieval 
using  phrases. 


[160]  Hughes,  M.  A.  (1997).  “Online  Documentation  in  Reference-based  Instruction:  A  Practical 
Model  for  Integrating  Help  Systems  into  Product  Training.”  Technical  Communication,  First 
Quarter:  58-64. 

Companies  can  improve  customer  satisfaction  while  reducing  training  time  and  product  support 
costs  by  integrating  online  documentation  with  product  training.  Online  documentation  can  be 
designed  to  be  not  only  the  reference  at  the  point  of  use  but  also  the  primary  instructional 
medium  used  during  training.  This  use  of  the  online  document  during  training  increases  user 
acceptance  of  it  and  helps  develop  the  required  skills  for  its  use.  This  expanded  role  for  online 
documentation  provides  new  opportunities  for  technical  communicators  to  add  value  to  their 
roles  within  their  companies.  This  article  defines  reference-based  instruction  and  outlines  its 
benefits.  It  describes  how  reference-based  instruction  can  be  incorporated  into  an  instructional 
system  design  (ISD)  and  provides  specific  examples  of  learning  objectives  and  student  exercises. 
It  lists  guidelines  for  how  to  structure  usability  tests  for  Help  systems,  and  finally,  it  advises  how 
technical  communicators  can  use  reference-based  instruction  to  expand  their  perceived  value  in 
their  organizations. 


[161]  McClintock,  M.  (1995).  Electronic  Text.  Personal  Technical  Memorandum. 

This  memo  is  a  brief  review  of  current  research  on  reading  text  from  VDU  screens.  Instead  of  a 
complete  review  of  the  literature,  this  memo  focuses  on  research  about  Preference  for  Paper, 
Display  Variables,  and  Interaction  Issues  (Scrolling  vs.  Paging,  Screen  Size,  Splitting  Text,  and 
Searching). 

[162]  McGee,  D.R.,  P.R.  Cohen,  R.M.  Wesson,  and  S.  Horman.  (April  20-25,  2002).  “Comparing 
Paper  and  Tangible,  Multimodal  Tools.”  Presented  at  CHI  2002,  Minneapolis,  MN,  407-414. 

In  command  posts,  officers  maintain  situational  awareness  using  paper  maps,  Post-it  notes,  and 
hand- written  annotations.  They  do  so  because  paper  is  robust  to  failure,  it  is  portable,  it  offers  a 
flexible  means  of  capturing  information,  it  has  ultra-high  resolution,  and  it  readily  supports  face- 
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to-face  collaboration.  The  authors  report  herein  on  an  evaluation  comparing  maps  and  Post-its 
with  a  tangible  multimodal  system  called  Rasa.  Rasa  augments  these  paper  tools  with  sensors, 
enabling  it  to  recognize  the  multimodal  language  (both  written  and  spoken)  that  naturally  occurs 
on  them.  In  this  study,  the  authors  found  that  not  only  do  users  prefer  Rasa  to  paper  alone,  they 
find  it  as  easy  or  easier  to  use  than  paper  tools.  Moreover,  Rasa  introduces  no  discernible 
overhead  in  its  operation  other  than  error  repair,  yet  grants  the  benefits  inherent  in  digital 
systems.  Finally,  subjects  confirmed  that  by  combining  physical  and  computational  tools.  Rasa  is 
resistant  to  computational  failure. 


[163]  Mitchell,  W.  J.  and  McCullogh,  M.  (1995).  Integrated  Design  Environments.  Digital  Design 
Media,  Second  Edition,  van  Nostrand  Rheinhold. 

An  integrated  computer-aided  design  environment  is  one  in  which  data  files  pertinent  to  a 
project,  software  tools,  and  physical  devices  can  efficiently  be  put  to  many  different  uses  - 
possibly  by  different  members  of  a  design  team,  working  in  different  places,  at  different  stages  in 
a  design  process  -  and  exploited  to  the  maximum.  Integration  is  achieved  by  carefully 
maintaining  the  currency,  integrity,  and  security  of  data  files  and  by  organizing  computer 
resources  so  that  these  files  and  appropriate  tools  for  manipulating  them  are  always  accessible 
when  and  where  they  are  needed  by  design  team  members  or  for  use  in  automatic  processes. 


[164]  Murphy,  A.  J.  (Ed.)  and  Engstrom,  D.  (1997).  “Book  Review:  A  Guide  to  Publishing  User 
Manuals  by  Ken  Whitaker.”  Technical  Communication,  Second  Quarter:  174-175. 

This  book  cover  of  Ken  Whitaker’s  A  Guide  to  Publishing  User  Manuals  describes  it  as  “A  total 
A-to-Z  guide  to  creating  professional-quality  user  manuals  on  time,  on  budget  -  on  your  PC.” 
Well,  the  book  doesn’t  quite  accomplish  that  goal.  But  it  does  offer  guidance  for  an  experienced 
writer  suddenly  thrust  into  the  role  of  page  and  document  designer.  The  book  touches  on  a  wide 
variety  of  topics  related  to  organizing,  writing,  producing,  and  printing  software  user 
documentation.  The  book’s  strongest  area  is  the  mechanics  of  desktop  publishing.  In  many 
respects,  the  book  is  a  kind  of  super  manual  for  ErameMaker  and  Corel  Ventura.  The  examples 
do  an  excellent  job  of  demonstrating  how  to  organize  a  dizzying  array  of  features  into  tasks  that 
you  might  actually  want  to  do.  Experienced  writers  who  have  lost  their  graphic  designers  to 
downsizing  or  outsourcing  may  find  the  book  a  lifesaver  when  stumbling  through  that  first 
project  without  professional  assistance. 


[165]  Murphy,  A.  J.  (Ed.)  (1997).  “Book  Reviews:  Technical  Communication.”  Technical 
Communication,  Third  Quarter:  307-325. 

Twelve  Book  Reviews.  (1)  Substance  &  Style:  Instruction  and  Practice  in  Copyediting,  (2) 
Industrial-strength  SGME:  An  Introduction  to  Enterprise  Building,  (3)  The  Scientific  Voice,  (4) 
Wired  Style:  Principles  of  English  Usage  in  the  Digital  Age,  (5)  Designing  Business:  Multiple 
Media,  Multiple  Disciplines,  (6)  Writing  in  a  Milieu  of  Utility:  The  Move  to  Technical 
Communication  in  American  Engineering  Programs,  1850-1950,  (7)  Digital  Type  Design  Guide: 
The  Page  Designer’s  Guide  to  Working  with  Type,  (8)  Dictionary  of  PC  Hardware  and  Data 
Communications  Terms,  (9)  The  Computer  Contradictionary,  (10)  The  Wired  Neighborhood, 
(11)  Shamans,  Software,  and  Spleens:  Eaw  and  the  Construction  of  the  Information  Society,  (12) 
SGME:  The  Billion  Dollar  Secret. 
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[166]  Muter,  P.  and  Maurutto,  P.  (1991).  “Reading  and  Skimming  From  Computer  Screens  and  Books: 
The  Paperless  Office  Revisited?”  Behaviour  and  Information  Technology  10(4):  257-266. 

Past  research  has  demonstrated  that  reading  efficiency  is  lower  from  the  standard  computer 
displays  on  the  1980s  than  from  paper.  In  the  present  experiments,  subjects  read  or  skimmed 
stories,  sometimes  from  a  high-quality  CRT  (cathode  ray  tube)  and  sometimes  from  a  book. 
Skimming  was  41%  slower  from  the  CRTs  than  from  the  book.  Possible  reasons  for  this  finding 
are  discussed.  Reading  speed  and  comprehension  were  equivalent  for  the  high-quality  CRTs  and 
the  book.  The  paperless  office  may  be  imminent  after  all. 


[167]  Osborne,  J.  (1995).  “Paperless  Wafer  Fabs.”  European  Semiconductor:  12-13. 

Until  recently,  semiconductor  fab  managers  had  few  alternatives  to  traditional  paper  manuals  for 
accessing  critical  information  on  system  operation,  troubleshooting,  and  maintenance.  With 
space  at  a  premium  and  paper  prohibited  in  the  cleanroom  due  to  particle  concerns,  technicians 
and  engineers  have  been  forced  to  physically  leave  the  cleanroom  each  time  they  needed  to 
consult  an  equipment  manual.  In  many  fabs,  managers  have  also  resorted  to  laminating  the  pages 
of  entire  maintenance  manuals  in  order  to  bring  sections  into  the  cleanroom.  Needless  to  say, 
neither  procedure  is  acceptable  in  today’s  wafer  fab  production  environment,  where  cost  of 
ownership  is  a  key  issue.  Equipment  makers  who  are  sensitive  to  cost  of  ownership  variables 
such  as  mean-time-to-repair  (MTTR)  and  mean-time-to-clean  (MTTC)  are  taking  advantage  of 
the  latest  in  multimedia  and  portable  computing  technologies  to  enhance  access  to  critical 
equipment  information  and  troubleshooting  capabilities.  In  most  cases,  the  technology  of  choice 
involves  the  conversion  of  hard  copy  manuals  to  cleanroom-compatible  electronic  media, 
notably  multimedia-capable  CD-ROMs,  which  can  easily  be  brought  into  the  cleanroom  by 
means  of  a  notebook  computer,  giving  engineers  access  to  vast  amounts  of  critical  data  on 
demand. 

[168]  Oviatt,  S.  (November  1999).  “Ten  Myths  of  Multimodal  Interaction.”  Communications  of  the 
ACM  42  (11). 

http://www.cse.ogi.edu/CHCC/Papers/sharonPaper/Myths/myths.html 

Multimodal  systems  process  combined  natural  input  modes-  such  as  speech,  pen,  touch,  manual 
gestures,  gaze,  and  head  and  body  movements-  in  a  coordinated  manner  with  multimedia  system 
output.  These  systems  represent  a  new  direction  for  computing  that  draws  from  novel  input  and 
output  technologies  currently  becoming  available.  In  this  article  ten  "myths"  about  multimodal 
interaction  are  identified  as  currently  fashionable  among  computationalists,  and  they  are 
discussed  from  the  perspective  of  contrary  empirical  evidence.  Current  information  is 
summarized  about  multimodal  interaction  from  research  on  multimodal  human-computer 
interaction,  and  from  the  linguistics  literature  on  natural  multimodal  communication.  In  the 
process  of  uncovering  misconceptions  associated  with  each  myth,  information  is  highlighted  on 
multimodal  integration  patterns  and  their  temporal  synchrony,  the  information  carried  by 
different  input  modes,  the  processability  of  users'  multimodal  language,  differences  among  users 
in  multimodal  integration  patterns,  and  the  reliability  and  other  general  advantages  of 
multimodal  system  design.  This  state-of-the-art  information  is  designed  to  replace  popularized 
myths  with  a  more  accurate  foundation  for  guiding  the  design  of  next-generation  multimodal 
systems. 


©  Copyright  2002,  Carnegie  Mellon  University 
October  2002 


54 


[169]  Paris,  C.  and  Linden,  K.V.  (1996).  “An  Interactive  Support  Tool  for  Writing  Multilingual 
Manuals.”  Computer  29(7):  49-56. 

Multilingual  instructional  manuals  are  important  for  companies  throughout  the  world  that  have  a 
significant  volume  of  international  business.  Using  automation  in  the  production  of  these 
manuals  is  attractive  because  of  the  increased  need  for  manuals  and  because  of  the  cost  and  time 
involved  in  creating  them. 


[170]  Redish,  J.  C.  (1997).  “Book  Review:  Dynamics  in  Document  Design:  Creating  Texts  for 
Readers  by  Karen  A.  Schriver.”  Technical  Communication,  Fourth  Quarter:  430-434. 

Dynamics  in  Document  Design  is  several  books  in  one  -  part  history,  part  detailed  information 
on  typography  and  layout,  part  research  and  theory,  part  case  studies  -  all  presented  in  clear 
writing  with  lots  of  illustrative  examples. 

[171]  Roccetti,  M.,  and  Salomon!,  P.  (2001).  “A  Web-based  Synchronized  Multimedia  System  for 
Distance  Education.”  ACM,  SAC  2001,  pgs.  94-98. 

The  authors  have  developed  a  Web  based  multimedia  educational  system,  which  has  been 
designed  to  provide  support  for  both  synchronous  and  asynchronous  distance  learning.  On  the 
one  hand,  the  system  enables  the  instructors  to  produce  and  the  students  to  receive  asynchronous 
customized  educational  material  under  the  different  form  of  HTML  slides,  digital  audio/video 
lectures,  software  packages  and  simulating  instruments  for  laboratory  activities.  On  the  other 
hand,  the  system  allows  users  (students  and  educators)  to  establish  synchronous  communications 
through  real  time  audio  connections.  In  the  paper,  the  authors  show  how  by  exploiting  the 
potentiality  of  adaptive  techniques,  markup  languages,  and  networked  multimedia  technologies, 
their  educational  tool  is  able  to  provide  each  different  user  with  an  educational  support  tailored 
to  his/her  personal  profile  and  didactical  needs. 


[172]  Schriver,  K.  A.  (1989).  “Document  Design  from  1980  to  1989:  Challenges  That  Remain.” 
Technical  Communication,  Fourth  Quarter:  313-331. 

Document  design  had  its  origins  in  the  1930s,  but  much  of  its  development  in  theory,  research, 
and  practice  has  occurred  in  the  past  ten  years  (1980-1989).  This  article  provides  a  snapshot  of 
the  evolution  of  document  design,  includes  a  comprehensive  list  of  research  references,  and 
stresses  the  need  to  integrate  theory  and  research  with  practice  as  we  move  into  the  1990s.  The 
author  concludes  by  identifying  critical  challenges  to  the  international  growth  of  document 
design,  both  in  the  academy  and  in  industry  -  challenges  that  must  be  met  if  document  design  is 
to  flourish. 


[173]  Shlaer,  S.  and  Mellor,  S.J.  (1989).  Object-Oriented  Systems  Analysis:  Modeling  the  World  in 
Data.  Englewood  Cliffs,  New  Jersey,  Yourdon  Press,  PTR  Prentice  Hall. 

Why  is  it  so  difficult  to  build  a  large  information  processing  system:  to  know  exactly  what  we 
are  doing  as  we  progress  through  the  many  steps  of  the  software  development  and  to  have  at  the 
end  (on  time  and  within  budget,  of  course)  a  system  that  meets  the  user’s  requirements  and 
expectations  and  is  easy  to  maintain,  modify,  and  understand?  On  the  basis  of  experience,  we 
have  identified  some  problems  that  seem  to  be  fundamental:  They  arise  again  and  again;  they 
plague  both  large  and  small  development  projects.  In  hindsight,  they  can  frequently  be  seen  as 
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major  contributors  to  project  failure.  These  problems  are  all  based  on  information  or 
misinformation  in  one  way  or  another. 

[174]  Shoemaker,  G.B.D.,  and  Inkpen,  K.M.  (March  31,  2001).  “Single  Display  Privacyware: 
Augmenting  Public  Displays  with  Private  Information.”  ACM,  SlGCHl  ’01,  3:  522-529. 

The  research  area  of  Single  Display  Groupware  (SDG)  confronts  the  standard  model  of 
computing  interaction,  one  user  working  on  one  computer,  by  investigating  how  to  best  support 
groups  of  users  interacting  with  a  shared  display.  One  problem  that  has  arisen  in  SDG  research 
concerns  access  to  private  information.  Previously,  private  information  could  not  be  displayed  on 
a  shared  display,  it  could  only  be  accessed  on  external  devices,  such  as  private  monitors  or 
Personal  Digital  Assistants  (PDAs).  This  paper  discusses  Single  Display  Privacyware  (SDP),  an 
interaction  technique  that  allows  private  information  to  be  shown  within  the  context  of  a  shared 
display.  A  description  of  the  hardware  and  software  components  of  our  prototype  SDP  system  is 
given.  The  results  of  a  user  study  using  a  prototype  SDP  system  indicated  that  SDP  is  a  valid 
interface  technique  that  is  well-accepted  by  users.  Only  one  of  the  sixteen  subjects  indicated 
discomfort  working  with  the  SDP  system.  This  was  a  first  attempt  at  choosing  a  domain  and  task 
suitable  for  a  SDP  system,  as  well  as  a  first  attempt  at  implementing  such  a  system.  It  was 
observed  that  different  users  have  different  preferences  concerning  what  information  should  be 
kept  private,  and  what  should  be  made  public. 

[175]  Smailagic,  A.,  D.P.  Siewiorek,  J.  Anhalt,  F.  Gemperle,  D.  Salber,  S.  Weber,  J.  Beck,  and  J. 
Jennings.  (May/June  2001).  “Towards  Context  Aware  Computing  Experiences  and  Lessons.” 
IEEE  Intelligent  Systems,  pgs.  38-46. 

This  paper  defines  an  activity/attention  framework  for  context  aware  computing,  and  categorizes 
several  applications  the  authors  have  developed  in  spatial  and  temporal  contexts.  These  context 
aware  applications  utilize  the  services  from  the  activity/attention  framework.  The  paper  also 
introduces  a  generic  architecture  for  pervasive  computing,  explores  and  refines  its  design  space, 
and  describes  specific  instantiations  of  the  architecture  and  provides  initial  evaluation  of  these 
applications.  Two  different  user  interfaces,  visual  and  audio,  have  been  implemented  and 
contrasted  for  the  same  application. 


[176]  Trego,  L.  (1996).  “Electronic  Tech  Manuals.”  Aerospace  Engineering:  pg  16. 

Delta  Air  Lines  has  adopted  the  software  of  Jouve  Data  Management,  along  with  the  services  of 
TransQest  Information  Solutions,  Inc.  for  distributing  airframe  technical  manuals  in  digital  form 
for  the  B-757,  B-767,  and  MD-1 1  fleets.  Delta  has  implemented  the  Jouve  Pinpoint  for  Windows 
retrieval  application  for  its  Atlanta  line  maintenance  workstations,  and  has  plans  to  install  the 
application  in  its  maintenance  hangars  and  engineering  facilities  in  Atlanta  and  at  line  stations 
throughout  its  system.  Data  will  be  distributed  via  LAN  in  seven  Delta  hub  cities,  with  stand¬ 
alone  CD-ROM  stations  installed  in  other  locations.  A  total  of  250  workstations  will  be  installed 
by  June  1996. 

[177]  von  Koenigseck,  E.  Irvin,  J.N.  and  Irvin,  S.C.  (1991).  Technical  Writing  for  Private  Industry: 
The  A-to-Z  of  O&M  Manuals.  Malabar,  Elorida,  Krieger  Publishing  Company. 

The  “A-to-Z  of  O&M  Manual  -  Technical  Writing  for  Private  Industry”  contains  three  parts.  Part 
I  describes  the  process  of  creating  the  manual;  Part  II  describes  the  typical  chapters  and  drawings 
in  the  manual;  Part  III  describes  the  subcontracting  process  and  databased  technical  manuals. 
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[178]  Whitefield,  A.  and  Warren,  C.  (1989).  “A  Blackboard  Framework  for  Modeling  Designers’ 
Behavior.”  Design  Studies  10(3):  179-187. 

A  framework  for  modeling  designers’  behavior  is  presented  and  discussed.  After  considering  the 
empirical  evidence  of  designers’  behavior,  and  extant  models  of  it,  a  modeling  framework 
derived  from  artificial  intelligence  ‘blackboard’  systems  is  presented.  Two  particular  models 
within  this  framework  are  then  detailed,  based  on  empirical  studies  of  experienced  designers. 
The  paper  concludes  with  a  discussion  of  the  strengths  and  weaknesses  of  the  blackboard 
framework. 
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2.  GOVERNMENT  DOCUMENTS 

[179] - Advanced  User  Interface  for  lETMs. 

http://www.nawcsti.navv.mil/aietm.htm 

Electronic  technical  manuals  revolutionized  the  way  in  which  technical  information  is  used.  East 
information  retrieval,  economical  production  and  distribution,  and  minuscule  stowage 
requirements  offer  solutions  to  problems  associated  with  the  use  of  paper  technical  manuals. 
Interactive  electronic  technical  manuals  (lETMs)  extended  these  advantages  by  improving  the 
way  in  which  information  is  presented  to  the  user.  In  spite  of  these  well-acknowledged  benefits, 
current  lETMs  have  limitations  that  impact  their  effectiveness  within  a  maintenance 
environment:  Most  are  desktop  or  laptop  systems  physically  separated  from  the  equipment  being 
maintained;  The  technician  must  divert  attention  from  the  maintenance  task  to  use  the  lETM. 
Most  provide  static  information  rather  than  assisting  in  repair  and  maintenance.  Eike  their  paper 
counterparts,  current  lETMs  provide  only  a  maintenance  reference  source  rather  than  becoming 
an  integral  part  of  the  maintenance  process.  Designed  to  support  a  variety  of  users  regardless  of 
environment,  experience,  or  skill  level,  current  lETMs  do  not  automatically  adjust  to  different 
user  levels. 


[180] - Interactive  Electronic  Technical  Manuals  and  Hypertext  Markup  Eanguage,  Dayton  T. 

Brown  Inc.  Technical  Communications. 
http://www.davtontbrown.com/tc  ietm.html 

The  authors  assert  that  the  benefits  of  using  lETMs  to  navigate  rapidly  through  vast  amounts  of 
technical  information  are  widely  recognized  by  industry  and  the  Government  as  time  and  cost 
savers.  This  is  especially  true  when  maintaining  complex  systems  and  equipment.  lETMs 
employ  interactive  presentation  techniques,  hypertext  linking,  still  photo  capture,  full  motion 
video,  video  frame  grabber,  animated  diagrams  and  keyed  audio.  lETM’s  can  also  interface  with 
CBT  programs.  Dayton  T.  Brown,  Inc.  uses  the  most  popular  design  software  to  develop  lETM’s 
to  fulfill  specific  client  requirements  allowing  linear  and  hierarchy  based  navigation.  Our 
designers  develop  Class  I  through  V  lETMs  for  new  or  mature  programs  which  will  allow  data 
base  revisions  by  either  developer  or  client.  lETMs  are  written  to  CD  for  direct  utilization.  They 
can  be  down  loaded  to  hard  drives  or  accessed  via  the  internet  using  HTME. 


[181] - (1999).  lETM  User-Interaction  (“Eook-and-Eeel”)  Guidelines  (MIE-PRE-87268A). 

Carderock,  Maryland,  Naval  Surface  Warfare  Center:  12. 

BACKGROUND.  At  the  1998  DoD  Eogistics  Reform  Day  Symposium  held  at  the  Pentagon  on 
1  October  1998,  there  were  a  number  of  presentations  and  exhibits  on  Interactive  Electronic 
Technical  Manuals  (lETMs).  Individual  Services  and  several  Industrial  activities  displayed  a 
number  of  exhibits  showing  state-of-the-  art  lETM  Presentation  Systems.  In  separate  formal 
presentations,  both  the  Assistant  Deputy  Undersecretary  of  Defense  (Eogistics  Reinvention)  and 
the  Head  of  Technology  and  Business  Integration  of  the  Joint  Electronic  Commerce  Project 
Office  (JECPO)  discussed  efforts  underway  to  achieve  DoD-wide  lETM  Interoperability.  At  that 
Eorum,  a  number  of  the  participants  suggested  that  an  area  in  need  of  additional  standardization 
was  that  of  lETM  user- interaction  features;  i.e.  the  look-and-feel  of  lETMs.  At  that  time, 
representatives  of  the  Aerospace  Industries  Association  (AIA)  Service  Publications  Committee 
and  of  the  Tri-Service  lETM  Technology  Working  Group  (lETMTWG)  agreed  it  would  be 
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desirable  to  hold  a  jointly  sponsored  Workshop  to  address  this  topic.  The  lETM  User- Interaction 
(“Look-and-Feel”  )  Workshop  resulting  from  this  agreement  was  held  at  the  Naval  Surface 
Warfare  Center  Carderock  Division,  West  Bethesda,  Maryland  on  15-18  March  1999.  Invitees 
included  representatives  selected  from  the  membership  of  the  AIA  Service  Publications 
Committee  and  two  representatives  each  from  the  Army,  Navy  ,  Air  Force  and  Marine  Corps. 
This  Report  summarizes  results  of  that  Workshop  and  provides  recommendations  for  further 
efforts  to  achieve  DoD-wide  standardization  of  lETMs,  based  on  a  common  look-and-feel 
approach. 


[182] - (2000).  Department  of  Defense  Handbook  for  Interoperability  of  Interactive  Electronic 

Technical  Manuals  (lETMs).  Redstone  Arsenal,  AE  35898-7466,  USAMC  Eogistics  Support 
Activity:  72. 

SCOPE.  The  purpose  of  this  handbook  is  to  outline  issues  associated  with  achieving  lETM 
interoperability  through  the  use  of  a  common  user  interface,  i.e.  a  browser.  Not  all  areas  of 
interoperability,  i.e.  data  interoperability  are  covered  in  this  handbook.  The  guidance  contained 
herein  specifically  covers  issues  that  may  allow  an  lETM  user  access  to  lETMs  via  a  common 
interface  regardless  of  where,  by  who,  and  how  the  lETM  was  created. 


[183] - (1998).  lETM  Overview,  Antech  Systems  (Specialists  in  Developmental  Technology 

and  Applications  for  Interactive  Presentation  of  Technical  Information  on  Digital  Media) 
www.antechsvstems.com/ 

An  lETM  Overview  including  these  five  topics:  lETM  Advantages.  Providing  the  Right  Data. 
lETM  Software  and  Database  Design.  Standardizing  lETMs.  lETM  Acquisition. 


[184] - Navy  lETM  Web  Site  (29  Einks). 

http://navvsgml.dt.navv.mil/ietm/webstuff/Reports.htm 


[185]  Broemm,  W.  J.  Ellner,  P.M.  and  Eipp,  J.  (1996).  A  Case  Study  in  the  Use  of  Reliability 

Projection  Methodology  to  Assess  Technical  Manuals.  Proceedings  -  Institute  of  Environmental 
Sciences. 

Whether  or  not  it  is  initially  planned  that  way,  reliability  testing  becomes  a  growth  process  as 
system  weaknesses  are  revealed.  Corrective  actions  (termed  fixes)  are  incorporated  both  during 
and  at  the  end  of  testing.  A  major  responsibility  of  reliability  engineers  is  to  assess  the  impact  on 
system  reliability  of  planned  or  implemented  fixes  to  problem  failure  modes  encountered  during 
testing.  This  reliability  assessment  should  provide  input  for  management’s  decision  with  regard 
to  whether  the  system  is  mature  enough  to  exit  the  current  text  phase  and  enter  the  next  test 
phase,  e.g.  exit  the  Pre-Production  Qualification  Test  (PPQT)  phase  and  enter  the  Initial 
Operational  Test  and  Evaluation  (lOT&E)  phase.  Projection  methodology  has  been  developed 
for  making  such  system  reliability  hardware/software  assessments.  Eor  the  system  considered  in 
this  paper,  the  technical  manuals  (TM’s)  were  thought  to  be  a  major  area  for  potential  system 
improvement  during  the  PPQT  phase.  This  raised  the  general  question  as  to  whether  reliability 
projection  methodology  could  provide  useful,  quantitative  measures  of  technical  manual 
maturity  and  adequacy. 
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This  paper  presents  a  case  study  based  on  TM  deficiencies  observed  during  the  PPQT  phase  for 
the  mobile  mass  spectrometer  (MM-1)  subsystem  of  the  FOX  System  -  a  nuclear,  biological, 
chemical  reconnaissance  system.  The  MM-1  is  the  primary  subsystem  used  for  detection  of 
chemical  agents.  This  subsystem  was  fully  mature  as  far  as  the  hardware  and  software  were 
concerned,  and  any  failures  of  this  nature  were  not  candidates  for  redesign  or  reprogramming. 
Any  improvement  in  MM-1  reliability  would  have  to  come  from  other  sources  such  as  training, 
manuals,  maintenance  and  test  equipment.  In  the  reliability  literature,  deficiencies  found  during 
testing  that  are  not  candidates  for  improvement  are  referred  to  as  A-modes,  and  those  that  are 
candidates  are  referred  to  as  B-modes.  In  this  paper,  the  proposed  projection  methodology  shall 
be  applied  to  a  subcategory  of  the  B-modes,  terms  B1  -modes,  associated  with  the  TM’s  for  the 
MM-1  subsystem.  Two  types  of  improvements  to  the  TM’s  were  performed  during  extensive 
testing  by  the  U.S.  Army.  The  first  concerned  changes  to  the  TM’s  due  to  errors  in  interpretation 
of  the  translation  from  German  to  English  as  well  as  operating  procedures  which  were  never 
placed  in  the  German  TM’s.  For  the  purposes  of  this  paper,  these  types  of  corrections  to  the 
TM’s  were  applied  to  deficiencies  that  we  shall  term  B1  failure  modes.  The  other  type  of 
improvement  to  the  TM’s  pertain  to  B-modes  caused  by  either  customary  practices  or 
requirements.  Changes  that  were  made  to  help  prevent  failures  from  these  causes  were  also 
placed  in  the  TM’s.  We  shall  refer  to  these  deficiencies  as  B2  failure  modes.  The  distinctions 
between  these  modes  are  further  discussed  in  the  paper.  Additional  test  information  was  obtained 
during  the  follow-on  lOT&E.  We  shall  use  the  lOT&E  data  to  motivate  our  discussion  of  several 
issues  that  arise  when  attempting  to  apply  reliability  projection  methodology  to  assess  TM’s. 


[186]  Dundas,  M.  Bigg,  K.  and  Peckham,  C.  (1999).  Electronic  Technical  Manuals. 
http :  //lrc3 .  monmouth .  army  .mil/cecom/lrc/pie/tm- ietm.html 

An  Electronic  Technical  Manual  (ETM)  is  a  digitized  and  electronically  presented  Technical 
Manual.  The  functionality  (level  of  interactivity)  and  structure  of  the  data  determines  the  type  or 
class  of  ETM.  An  Interactive  ETM  (lETM)  is  a  type  of  ETM  that  has  more  sophisticated  means 
of  interactivity  and  is  formatted  in  a  database.  This  topic  will  speak  generally  to  ETMs  with  the 
understanding  that  the  “I”  in  lETMs  is  based  on  selection  of  functionality/interactivity 
requirements  and  is  a  program  specific  requirement.  The  Army  is  transitioning  to  acquiring 
ETMs  developed  to  performance-based  specifications  and  standards.  The  intent  of  this  transition 
is  to  position  ourselves  for  maximum  use  of  the  commercial  technological  base,  wherever 
possible,  in  the  development  of  our  products.  To  accomplish  this  will  require  the  development  of 
acquisition  requirements  that  enables  the  use  of  the  most  current  publishing  and  delivery 
technology,  without  restricting  its  future  reuse  as  legacy  data.  Maximum  use  will  be  made  of 
resources  and  information  furnished  in  the  form  of  Intelligent  Product  Data  (IPD).  This  will 
leverage  cost  paid  to  own/use/access  this  data  and  eliminate  paying  for  redundant  submittals  of 
the  same  data  to  separate  functional  areas. 


[187]  Fuller,  J.  J.  (1998).  Plan  for  DoD  Wide  Demonstrations  of  a  DoD  Improved  Interactive 
Electronic  Technical  Manual  (lETM)  Architecture.  Carderock  Division,  West  Bethesda, 
Maryland,  Naval  Surface  Warfare  Center:  84  pgs. 

This  Report  describes  the  Pilot-Demonstration  Phase  of  the  development  of  a  Joint  lETM 
Architecture  (JIA)  designed  to  assure  Service-wide  interoperability  of  Interactive  Electronic 
Technical  Manuals  (lETMs)  based  on  emerging  World  Wide  Web  technology.  The  goal  of  the 
JIA  is  to  provide  an  environment  in  which  all  legacy,  and  newly  acquired.  Electronic  Technical 
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Manuals  can  be  read  by  any  end  user  with  a  common  user-interface  display  system,  regardless  of 
authorship  of  the  Technieal  Information.  The  JIA  development  program  is  being  earried  out  by 
the  Tri-Serviee  lETM  Teehnology  Working  Group  (lETMTWG),  ehaired  by  the  Naval  Surface 
Warfare  Center,  Carderock  Division  (NSWCCD),  in  accordance  with  directives  established  by 
the  Office  of  the  Assistant  Deputy  Undersecretary  of  Defense  (ADUSD(ERM))  and  the  Joint 
Commanders  Group  for  Communieations  and  Eleetronics  (JCG-CE)  of  the  Joint  Eogistics 
Commanders.  The  development  effort  will  culminate  in  the  promulgation  of  appropriate  policy 
directives,  user-guidance  documentation,  and  appropriate  MIE-PRE-acquisition  documents  to 
assure  full-spectrum  interoperability.  The  Pilot  Demonstration  Phase  of  the  effort  has  been 
established  to  demonstrate  that  representative  systems  of  the  Army,  Navy,  Air  Eorce,  and  Marine 
Corps  ean  successfully  be  made  JIA-compliant  with  a  minimum  of  effort.  The  demonstration 
systems  chosen  for  this  purpose  by  Service  members  of  the  lETMTWG  include  a  complete  range 
of  lETM  complexity,  from  the  simple  forms  of  digitized  legacy  Technical  Information  to  the 
most  highly  developed  and  integrated  interactive  Technieal  Information  systems.  Similarly,  they 
include  a  broad  speetrum  of  commereial  technologies  now  in  use  for  generation  of  lETMs. 


[188]  Euller,  J.  J.  (1997).  Tri-Service  Interactive  Eleetronic  Technical  Manual  (lETM)  Technology 
Working  Group.  Naval  Sea  Systems  Command.  Naval  Surface  Warfare  Center:  Carderock 
Division.  CAES  Expo  International  1997. 

Slide  presentation  consists  of:  Working  Group,  Purpose,  Membership,  Responsibilities,  PY97 
Teehnieal  Initiatives,  Points  of  Contact,  What  are  lETMs?,  NSWC/CD  R&D  in  70’ s  and  80’ s 
Reading  to  lETMs,  lETM  Payoffs. 


[189]  Puller,  J.  J.  EeBeau,  R.P.  Mavor,  A.S.  Post,  T.J.  and  Sawyer,  C.R.  (1988).  Test  and  Evaluation 
of  the  Navy  Technical  Information  Presentation  Systems  (NTIPS)  AN/SPA-25D  Pield  Test 
Results.  Bethesda,  MD,  David  Taylor  Research  Center:  89  pgs. 

This  doeument  describes  the  results  of  the  second  Pield  Test  of  Navy  System-related  Technieal 
Information  (TI)  developed  by  the  Navy  Technical  Information  Presentation  System  (NTIPS). 
Three  types  of  experimental  TI  were  compared  with  the  conventional  Technical  Manual.  Por 
performing  troubleshooting  tasks,  the  NTIPS  eleetronically  displayed  automated  troubleshooting 
TI  (ealled  Pault  Isolation  by  Nodal  Dependency,  PIND)  was  compared  with  the  conventional 
Technical  Manual,  and  for  performing  corrective  maintenance,  NTIPS  electronically  displayed 
TI  was  compared  with  NTIPS  on  paper  and  the  conventional  Teehnieal  Manual.  Tests  were 
earried  out  at  the  Naval  Sea  Combat  Systems  Engineering  Station  in  Norfolk,  VA.  using  two 
operational  AN/SPA-25D  radar  repeaters  (with  introduced  faults).  Test  subjeets  were  Eleetronic 
Technicians  stationed  on  ships  and  at  shore  based  facilities  in  Norfolk,  VA.  All  test  objectives 
were  achieved.  Almost  all  (92%)  of  the  subjects  preferred  electronically  delivered  TI  to  the 
conventional  Technical  Manual.  They  were  able  to  use  NTIPS  TI  to  troubleshoot  more 
aceurately  and  with  greater  speed  than  with  the  eonventional  Technieal  Manual.  Por  corrective 
maintenance,  subjects  performed  at  the  same  accuracy  and  speed  regardless  of  the  TI  type. 
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[190]  Fuller,  J.  J.  Post,  TJ.  and  Mavor,  A.S.  (1988).  Test  and  Evaluation  of  the  Navy  Technical 
Information  Presentation  System  (NTIPS)  F-14A  Field  Test  Results.  Bethesda,  MD,  David 
Taylor  Research  Center:  94  pgs. 

This  document  describes  the  results  of  one  part  of  a  full-scale  system  test  of  concepts  for 
improvement  of  quality  and  of  display  of  Navy  System-related  Technical  Information  developed 
by  the  Navy  Technical  Information  Presentation  System  (NTIPS).  The  tests  involved 
comparisons  of  three  types  of  experimental  Technical  Information  (TI)  with  the  conventional 
paper  work  package.  The  three  types  of  experimental  TI  were  (1)  NTIPS  automated 
troubleshooting  TI  (called  Fault  Isolation  by  Nodal  Dependency,  FIND),  electronically 
displayed;  (2)  NTIPS  electronically  displayed  corrective-maintenance  TI;  and  (3)  NTIPS 
corrective-maintenance  TI  delivered  on  paper.  Tests  were  carried  out  at  Miramar  Naval  Air 
Station  using  an  operational  F-14A  aircraft  (with  introduced  “faults”).  Test  subjects  were 
squadron  Aviation  Electricians  Mates  (AEs)  both  experienced  and  inexperienced.  All  test 
subjects  were  achieved.  The  test  demonstrated  that  electronic  display  of  maintenance  TI  is  highly 
acceptable  to  fleet  personnel  (90%  of  the  test  subjects  favored  electronic  display),  that  NTIPS- 
proposed  modifications  to  TI  are  as  effective  or  more  effective  than  conventional  TI  in 
supporting  troubleshooting  and  corrective  maintenance,  and  that  automated  troubleshooting 
produced  a  highly  significant  improvement  in  fault-isolation  success.  The  test  provided  valuable 
experience  and  indicated  areas  in  which  NTIPS  and  electronic  display  can  be  improved. 
Proposals  resulting  from  the  test  are  currently  being  implemented. 


[191]  Fuller,  J.  J.  (October  1994).  lETMs:  From  Research  to  Reality,  Naval  Surface  Warfare  Center, 
Carderock  Division.  AFEI  CAES  Expo  International  1994. 
http://navvsgml.dt.navv.mil/ietm/ietm.html 

In  the  late  1970s,  the  U.S.  Military  Services  formulated  concepts  for  Interactive  Electronic 
Technical  Manuals  (lETMs)  to  replace  Technical  Manuals  (TMs)  presented  on  paper  and 
microform.  Comprehensive  research  and  development  programs  were  conducing,  including 
Navy  Technical  Information  Presentation  System  (NTIPS)  and  the  Air  Force  Computer-based 
Maintenance  Aid  System  (CMAS).  In  the  1980s,  pilot  systems  were  developed  and  tested  under 
operational  conditions.  Significant  quantitative  payoffs  were  demonstrated,  with  overwhelming 
field-user  preference  for  lETMs  over  paper-based  TMs.  Based  on  these  successes,  the  Joint 
Industry/Government  Pageless  TM  Committee  was  formed  and  worked  to  standardize  lETM 
approaches  and  technology.  The  Tri-Service  lETM  Working  Group  developed  DoD 
specifications  for  the  acquisition  of  lETMs;  and  the  CAES  ISG  Standards  Division  reviewed  and 
concurred  with  these  specifications.  lETM  authoring  and  presentation  systems  have  emerged  in 
the  commercial  marketplace.  In  the  1990s,  DoD  programs  are  acquiring  lETMs  to  support 
weapon  systems,  such  as  Paladin,  Apache,  Comanche,  AEGIS,  EDS,  BSY-2,  F-22,  JSTARS,  and 
V-22.  Commercial  applications  are  underway  in  the  airline,  automotive,  and  railroad  industries. 
Internationally,  lETMs  are  proposed  for  the  NATO  NH-90  helicopter  and  the  European  Fighter 
Aircraft.  Thus,  lETMs  have  progressed  from  concepts,  through  pilot  development,  field  testing, 
and  standardization,  to  military  and  commercial  implementations.  Clearly,  lETMs  have  moved 
“From  Research  to  Reality.” 
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[192]]  Harvey,  B.  (May  1998)  How  STEP/SGML  and  XML/EDI  Initiatives  Will  Impact  Euture 

Development  of  lETMs.  Electronic  Commerce  Connection,  Inc.  Paper  presented  at  XME  Europe 

Conference  in  Paris,  Prance. 

http :  //w  w  w .  eccnet.com/papers/ietm.html 

The  STEP/SGME  initiative  will  enable  information  objects  used  within  technical  manuals  to  be 
created  early  in  the  manufacturing  process  and  then  used  within  paper  manuals.  Interactive 
Electronic  Technical  Manuals  (lETMs),  Computer  Based  Training  (CBT),  documents  on 
demand,  etc.  This  paper  will  describe  how  the  ISO  Preliminary  Work  Item  (PWI)  on  SGME  and 
Industrial  Data,  and  recent  initiatives  towards  using  extensible  Markup  Eanguage  (XME)  for 
Electronic  Data  Interchange  (EDI),  could  ultimately  impact  and  improve  the  lETM  authoring 
and  creation  process. 


[193]  Harvey,  B.  (1997).  “Interactive  Electronic  Technical  Manuals  and  STEP.”  1-39. 

ISO  STEP  Conference,  Chester,  England  -  Slide  Presentation.  Slide  material  covers:  Definition 
of  lETMs  (History,  Concepts,  and  Specifications),  How  SGME/HyTime  is  used  for  lETMs,  and 
How  can  STEP  help  in  development  of  lETMs? 


[194]  Jorgensen,  E.E.  (1999).  NSWCCD-20-TR-1999-12-I-TR  Proposed  DoD  Guidelines  for 

Implementation  of  a  Web-Based  Joint  lETM  Architecture  (JIA)  to  Assure  the  Interoperability  of 
DoD  lETMs.  West  Bethesda,  MD,  Carderock  Division,  Naval  Surface  Warfare  Center:  58  pgs. 
This  paper  presents  preliminary  guidelines  intended  to  serve  as  input  to  a  planned  DoD 
Handbook  for  the  Acquisition  and  Deployment  of  DoD  lETMs  with  the  specific  purpose  of 
assuring  interoperability  of  lETMs  among  various  end-users. 


[195]  Jorgensen,  E.E.  (1998).  CDNSWC-TSSD-98-10  Proposed  Web-Based  Joint  lETM  Architecture 
(JIA)  for  the  Interoperability  of  DoD  lETMs.  West  Bethesda,  MD,  Carderock  Division,  Naval 
Surface  Warfare  Center:  33  pgs. 

This  report  documents  the  results  of  a  study  sponsored  by  the  Assistant  Deputy  Secretary  of 
Defense  for  Eogistics  Reinvention  and  Modernization  (ADUSD  (ER&M))  and  the  Joint 
Commanders  Group  for  Communications  and  Electronics  (JCG-CE).  An  Architecture  is 
proposed  for  Interactive  Electronic  Technical  Manuals  (lETMs)  based  on  the  technology, 
industry  standards,  and  commercial  software  products  being  developed  for  the  World  Wide  Web. 


[196]  Jorgensen,  E.E.  (1998).  CDNSWC-TSSD-98-005  Proposed  Web-Based  Architecture  for  the 
Interoperability  of  Naval  Aviation  lETMs.  West  Bethesda,  MD,  Carderock  Division,  Naval 
Surface  Warfare  Center:  32  pgs. 

This  report  documents  the  results  of  a  study  sponsored  by  the  Naval  Air  Systems  Command 
(NAVAIR)  which  is  recommending  an  Architecture  for  Naval  Aviation  Interactive  Electronic 
Technical  Manuals  (lETMs)  based  on  the  technology,  industry  standards,  and  commercial 
software  products  being  developed  for  the  World  Wide  Web. 
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[197]  Jorgensen,  E.L.  (1998).  The  Future  Benefits  of  Utilizing  a  Web-Based  Architecture  for  DoD 
lETMs,  CAES  EXPO  1998. 

Slide  presentation  consisting  of:  Developing  a  Web-Based  Approach  to  DoD  lETMs,  The  JIA 
Foundation,  Benefits  to  the  User,  Benefits  to  Authoring  Organization,  Future  Expandability,  Ad 
Hoc  Aggregations  of  lETMs,  Freebies  Gained  by  Going  Web,  Objective  Obtained  lETM 
Interoperability-i-,  Innovative  Concept  -  Using  Web  Objects  as  Source  Data,  Authoring  Process, 
User  Site  Process,  Future  Expansion,  and  Final  Thoughts. 


[198]  Jorgensen,  E.E.  (1997).  DoD  lETM  Interoperability  Project,  NSWC  Carderock  Division, 
Department  of  the  Navy. 

Slide  presentation  consisting  of:  DoD  lETM  Interoperability,  Background,  Issues  with  lETM 
Interoperability,  Results,  Proposed  Approach  to  Solving  the  Interoperability  Problem,  Selected 
General  Approach,  lETM  Interoperability,  Interoperability  applies  to  the  entire  lETM 
Community,  Authoring  Process,  Infrastructure  Process,  User  Site  Process,  User  Site 
Interoperability,  Specifications  Needed,  Initial  Development,  Projected  Future  Goals,  Include 
Interface  with  Other  Job  Performance  Aids,  Web-Based  Concept  of  Single  Product  Support  Data 
Base,  and  Summary. 


[199]  Jorgensen,  E.E.  (1996).  Achieving  Interoperability  Through  a  Proposed  New  Information 

Architecture  for  the  Acquisition  and  Use  of  lETMs.  Carderock  Division,  Naval  Surface  Warfare 
Center,  Department  of  the  Navy. 

Slide  presentations  consisting  of:  Background,  Issues  with  lETM  Interoperability,  Proposed 
Approach  to  Solving  the  Interoperability  Problem  for  Suppliers  and  Users,  Developments  in 
Web/Internet  Technology  used  in  Developing  the  Proposed  New  lETM  Process,  Why  Web 
Technology?  Overview  of  the  New  Proposed  lETM  Process,  How  Interoperability  is  Achieved, 
and  a  Disclaimer. 


[200]  Jorgensen,  E.E.  (May  1996)  The  Move  to  Paperless  Technical  Manuals  in  the  U.S.  DoD. 
Caderock  Division,  Naval  Surface  Warfare  Center,  U.S.  Department  of  the  Navy:  1-7.  CAES 
Europe. 

http://navvsgml.dt.navv.mil/ietm/ietm.html 

In  this  paper  Mr.  Jorgensen  attempted  to  categorize  and  describe  the  various  moves  in  the  US 
Department  of  Defense  from  paper-based  Technical  Manuals  to  a  variety  of  forms  of  Electronic 
Technical  Manuals  (ETMs).  The  three  principal  Services  have  agreed  to  a  categorization  of  five 
Classes  of  ETMs  and  this  discussion  will  follow  these  five  categories,  which  range  form  simple 
Class  1  “page-flippers”  to  a  complete  Class  5  Technical  Information  Systems  under 
consideration  which  utilize  a  common  data  base  derived  from  Eogistic  Support  Analysis  Records 
(ESAR)  and  support  Maintenance,  Operations,  Training,  and  Eogistics  Support  functions.  In 
between  these  two  extremes  are  the  Interactive  Electronic  Technical  Manuals  (lETMs)  specified 
by  the  two  US  Military  Specifications  MIE-M-87268  and  MIE-D-87269.  The  paper  concludes 
with  some  of  Mr.  Jorgensen’s  observations  and  opinions  on  the  possibility  of  obtaining 
consensus  on  International  Standards  to  guide  the  production  of  standard  paperless  and 
interactive  electronic  support  systems  which  can  be  used  across  International  borders. 
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[201]  Jorgensen,  E.L.  (1995).  Maturing,  Migrating,  Merging,  or  Mixing?  The  Future  of  the  DoD 
lETM  Specifications.  AFEI  CAES  Expo  International. 
http://navvsgml.dt.navv.mil/ietm/ietm.html 

Slide  presentation  consisting  of:  Original  Concept  to  Spec  lETMs,  The  DoD  lETM  Specs 
Roadmap,  Impact  of  the  “Perry  Memo”,  Current  Plans  for  Specifications,  Eegacy  Data 
Requirements,  Some  Recommendations,  and  Harmonization  with  Other  Standards. 


[202]  Jorgensen,  E.E.  (1994).  DoD  Classes  of  Electronic  Technical  Manuals.  Caderock  Division, 
Naval  Surface  Warfare  Center:  1-9. 

Over  the  last  two  or  three  years  the  concept  of  electronically  presenting  Technical  Manuals  to 
end-users  has  gone  from  an  experimental  concept  to  one  which  is  widely  regarded  as  an  idea 
whose  time  has  come.  Many  DoD  Acquisition  Program  Managers  have  already  begun  to 
implement  Automated  TM  systems  for  their  own  programs  because  they  make  sense.  However, 
these  applications  have  not  been  centrally  managed  or  developed  to  common  specifications,  and 
as  a  result  many  substantially  different  concepts  and  incompatible  systems  have  emerged  in 
demonstration  programs  and  actual  applications.  Some  of  these  are  now  being  proposed  as 
candidates  for  the  “Standard  System”  for  automated  TM  presentation  systems,  while  other 
approaches  are  quietly  being  implemented  as  “tactical  and  mission-critical  systems”  so  as  to 
remain  unimpeded  by  possible  non-tactical  standards  and  bureaucracies.  It  is  in  this  environment 
that  the  Navy  ATIS  System  as  well  as  the  new  DoD  lETM  Specifications  are  proposed  as  the 
basis  for  the  set  of  required  DoD  standards  needed  to  avoid  undesirable  non-compatible 
proliferation  of  automated  TM  systems. 


[203]  Jorgensen,  E.E.  (October  1994).  The  Interactive  Electronic  Technical  Manual  Overview 
“Setting  the  Stage”,  Carderock  Division,  Naval  Surface  Warfare  Center.  AFEI  CAES  Expo 
International  1994. 

http://navvsgml.dt.navv.mil/ietm/ietm.html 

The  paper  presents  a  general  overview  of  DoD  Interactive  Electronic  Technical  Manuals 
(lETMs)  and  opens  with  a  short  discussion  of  the  problems  leading  up  to  the  need  for  lETMs,  the 
general  characteristics  of  an  lETM,  and  an  identification  of  the  major  research  programs  in  the 
three  Services  leading  up  to  lETMs. 


[204]  Jorgensen,  E.E.  (1991).  DoD  Specifications  for  Interactive  Electronic  Technical  Manuals 
(lETM).  Status  Report  on  Draft  Specifications  and  Handbooks  Developed  by  the  Tri-Service 

Working  Group  for  lETMs,  Department  of  the  Navy. 

Presentation  slides  consisting  of:  Working-Group  Task  and  Membership,  Draft  Specifications, 
Current  Releases  and  Status,  Plans  for  FY-92,  and  Key  Issues. 


[205]  Jorgensen,  E.E.  and  Fuller,  J.J.  (October  19-21,  1998).  A  Web-Based  Architecture  for 

Interactive  Electronic  Technical  Manuals  (lETMs):  1-11.  ASNE  Naval  Eogistics  Conference. 
http://navvsgml.dt.navv.mil/ietm/ietm.html 

This  paper  presents  a  concept  of  a  Web-based  Architecture  for  achieving  user-level 
interoperability  of  DoD  Interactive  Electronic  Technical  Manuals  so  that  an  end  user  can  view 
any  DoD  lETM,  no  matter  what  the  source,  using  only  one  electronic  display  device  and 
common  set  of  browser  software.  The  particular  solution  presented  is  that  developed  by  a  study 
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performed  for  the  Navy;  however,  the  paper  also  discusses  the  effort  to  extend  the  Navy 
Architecture  to  all  of  DoD.  The  DoD  effort  is  being  undertaken  by  a  Tri-Service  team  chartered 
by  the  Assistant  Undersecretary  of  Defense  (Logistics  Reinvention  and  Modernization).  The 
Architecture  is  being  developed  in  response  to  a  requirement  from  the  Joint  Logistics 
Commanders,  which  identifies  that  non-interoperability  of  lETMs  as  a  major  impediment  in 
conducting  Joint  Operations. 


[206]  Jorgensen,  E.L.  and  Fuller,  J.J.  (1996).  New  Approaches  For  Navy  Technical  Training  And  Job 
Performance  Aiding  Using  Expanded  lETM  Technology.  NSWCCD/TSS-97-01.  Carderock 
Division,  Code  2052,  Naval  Surface  Warfare  Center:  1-24. 

As  early  as  the  1970’s,  the  Fleet  User  Systems  Branch  (2052)  of  the  Carderock  Division,  Naval 
Surface  Warfare  Center  (NSWC/CD)  began  to  develop  better  ways  to  present  Technical 
Information  (TI)  to  Navy  Fleet  users  needed  for  purposes  of  operating,  maintaining,  training,  and 
logistically  supporting  the  Navy’s  inventory  of  weapon  systems.  Prior  to  that  time,  with  the 
exception  of  Training  Films,  virtually  all  Technical  Information  was  based  on  paper  media.  This 
effort  resulted  in  the  formulation  of  new  concepts  and  capabilities  which  permitted  the 
replacement  of  traditional  paper  Technical  Manuals  with  the  Interactive  Electronic  Technical 
Manual  (lETM).  A  series  of  laboratory  and  field  tests  conducted  by  the  Air  Force  and  the  Navy 
in  the  late  1980’s  demonstrated  the  benefits  of  lETMs  for  performing  Organizational-Fevel 
maintenance.  Based  on  the  experience  and  success  of  those  initial  tests,  DoD  lETM 
Specifications  were  developed  by  a  Tri-Service  Working  Group  chaired  by  NSWC/CD.  The 
Specifications  were  initially  promulgated  in  November,  1992.  They  were  intentionally  very 
forward  looking  and  included  provisions  for  including  advanced  Object-Oriented  Technology 
and  Interactive  Multi-Media  Technology  as  they  matured  and  became  more  available  in  the 
future.  Since  that  time,  many  Navy  Programs  have  planned  for  and  developed  lETMs  in  one 
form  or  another,  some  conforming  closely  to  the  lETM  Specifications  and  others  using  emerging 
COTS  (Commercial-Off-the-Shelf)  electronic  document-viewing  products  when  the  legacy 
format  of  the  existing  paper-based  TMs  rendered  the  MIL-SPEC  lETMs  too  costly. 


[207]  Jorgenson,  E.F.  and  Fuller,  J.J.  (March  17-18,  1993).  The  Interactive  Electronic  Technical 
Manual.  ASNE/SOFE  Conference. 
http://navvsgml.dt.navv.mil/ietm/ietm.html 

Paper  consists  of  the  following  sections:  The  Existing  Technical  Manual  Problem,  Emerging 
Solutions,  The  lETM  Concept,  RDT&E  Reading  up  to  the  lETM  Concept,  Measurable  Benefits 
Resulting  from  Use  of  lETMs,  The  Development  of  lETM  Specifications  for  DoD,  Status  of  the 
DoD  lETM  Specifications,  and  The  Future  for  lETMs. 


[208]  Jorgensen,  E.F.  Fuller,  J.J.  Post,  T.J.  and  Rainey,  S.C.  (1995).  Potential  Benefits  To  Navy 
Training  Programs  Resulting  From  Increased  Use  of  Interactive  Electronic  Technical  Manuals. 
Carderock  Division,  Bethesda,  Maryland  20084-5000,  Naval  Surface  Warfare  Center:  130  pgs. 
This  report  summarizes  the  first  phase  of  a  study  entitled  Training  Benefit  Analysis  of  the 
Accelerated  Use  of  Interactive  Electronic  Technical  Manuals  (lETMs).  An  initial  evaluation  of 
the  potential  effectiveness  in  Navy  training  of  the  use  of  the  interactive,  computer-controlled 
display  of  Technical  Information  has  been  carried  out.  Results  indicate  a  widespread  agreement 
in  the  Navy  Training  Community  that  the  use  of  lETMs,  integrated  with  automated  courseware. 
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could  be  of  significant  benefit  in  producing  greater  effectiveness  in  training  procedures.  Fifteen 
lETM-use  “Hypotheses”,  together  with  associated  implementation  scenarios,  have  been 
evaluated  by  the  Training  Community.  In  a  detailed  review,  twelve  of  these  possible  applieations 
were  supported  by  more  than  two  thirds  of  the  participants;  some  by  more  than  90  percent. 
Forty-seven  Candidate  Projects  for  specific  applications  of  lETMs  in  Navy  training  use  have 
been  elieited  from  Surface,  Air,  and  Submarine  warfare  areas  of  the  Navy.  These  Candidate 
Projeets,  whieh  are  summarized  in  this  report,  have  been  reviewed  by  the  study  team  and  found 
to  provide  a  sound  basis  for  the  business-case-analysis  phase  of  the  study.  This  report  also 
summarizes  a  number  of  technological  and  administrative  issues,  identified  by  the  Training 
Community,  which  must  be  addressed  prior  to  realizing  the  full  potential  of  lETMs  in  Navy 
training.  Measures  needed  for  greater  integration,  infrastrueture  support,  and  standardization  of 
lETMs  in  training  are  recommended.  Phase  II  of  the  study  will  consist  of  a  more  detailed 
analysis  of  the  selected  Candidate  Projects,  particularly  from  the  standpoint  of  Return  on 
Investment,  to  provide  the  Chief  of  Naval  Operations  with  the  basis  for  a  training  input  to  the 
Program  Objeetives  Memorandum  for  EY  98  (POM  ’98)  preparation  proeess. 

[209]  Karabulut,  A.N.,  and  Oz,  H.  (December  1998).  “How  to  Develop  an  Interactive  Electronic 
Technical  Manual:  An  Industry  Perspective  -  Thesis.”  Air  Eorce  Institute  of  Technology, 
Department  of  the  Air  Force,  Air  University,  Wright-Patterson  Air  Eorce  Base,  Ohio,  126  pgs. 
Improvements  in  technology,  especially  in  computer  science,  in  the  last  two  decades  have  made 
it  possible,  and  preferable  to  develop  digital  technical  manuals.  A  digital  manual,  which  is  called 
an  Interaetive  Eleetronic  Technieal  Manual  (lETM),  is  a  package  of  information  required  for  the 
diagnosis  and  maintenance  of  a  weapon  system,  optimally  arranged  and  formatted  for  interaetive 
screen  presentation  to  the  end-user.  Being  the  largest  organization  in  the  U.S.,  the  Department  of 
Defense  has  pioneered  in  the  development  of  the  lETM  concept  as  well  as  in  the  establishment 
of  its  standards.  There  have  been  many  researches  done  about  different  lETM  applications  and 
their  effectiveness  in  the  DoD  environment.  However,  little  researeh  has  been  done  in  the  area  of 
how  an  lETM  is  developed  in  a  eivilian  environment.  This  thesis  identifies  what  it  takes  to 
develop  an  lETM  in  a  civilian  environment  and  investigates  differentiating  factors  of  the 
eommercial  industry.  In  addition  to  the  identification  of  lETM  development  steps  in  a  case  study, 
lETM  standards,  lETM  development  specifieations  in  industry  as  well  as  in  the  military, 
problem  areas  in  today’s  lETM  development  environment,  and  DoD  classification  of  lETMs  are 
also  discussed. 

[210]  Kramer,  M.T.  Post,  T.J.  and  Rainey,  S.C.  (1993).  Results  of  a  Joint  Navy/Air  Eoree  Operational 
Test  to  Evaluate  USAE  Integrated  Maintenance  Information  Systems  (IMIS)  Interaetive 
Eleetronic  Technical  Manual  (lETM)  Technology  Applied  to  the  E/A- 18  Aircraft.  Bethesda, 

MD,  Carderock  Division,  Naval  Surface  Warfare  Center:  143  pgs. 

In  a  joint  effort,  the  U.S.  Navy  and  Air  Eoree  have  tested  under  operational  conditions  a  series  of 
improved  teehniques  developed  under  the  Air  Eorce  Integrated  Maintenanee  Information  System 
(IMIS)  program  for  delivering  maintenance  teehnieal  information  to  squadron  teehnieians.  These 
improvements  included  use  of  a  portable  maintenance  aid  (PMA)  for  technical  information 
display.  In  this  test,  earried  out  in  an  E/A- 1 8  fighter  squadron  at  Marine  Corps  Air  Station, 
Beaufort,  South  Carolina,  each  of  16  technicians  performed  six  fault-isolation  tasks,  three 
supported  by  an  Interaetive  Eleetronic  Technical  Manual  (lETM)  displayed  on  a  PMA,  and  three 
supported  by  conventional  paper-based  work  package  (WP)  technical  manual  This  report 
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presents  a  comparison  of  technician  performance  supported  by  the  lETM/PMA  combination  with 
performance  using  conventional  paper  TMs.  The  performance  data  collected  during  the  test 
showed  considerable  reduction  in  performance  times  for  complex  multiple  fault  isolations  and 
maintenance  errors,  when  technicians  used  the  lETM/PMA  combination.  Also,  technician- 
preference  data  based  on  questionnaires,  showed  strong  support  for  virtually  all  lETM/PMA 
features.  Technicians  also  made  suggestions  for  improving  the  lETM  and  the  PMA.  The  test 
results  have  been  provided  to  both  Services  as  one  basis  for  evaluating  and  improving  these 
IMIS  technologies;  specifically,  (1)  the  specifically  developed  PMA;  (2)  preparation  and 
automation  of  the  lETM  technical  information  by  automated  methods  based  on  initial 
construction  of  an  lETM  data  base,  lETMDB;  (3)  the  IMIS  Presentation  System  (PS);  (4)  on¬ 
ground  fault  isolation  using  data  from  aircraft  Built-in  Test  Equipment  (BITE)  coupled  directly 
to  the  PMA  through  a  1553  bus;  (5)  the  IMIS  Diagnostic  Module  (DM),  and  (6)  the  IMIS  Human 
Computer  Interface  (HCI)  module.  The  lETMDB  and  the  lETM  technical  information  were 
constructed  generally  in  accordance  with  the  new  DoD  Specifications  MIE-D- 87269  and  MIE- 
M-87268,  respectively. 

[211]  Eane,  K.W.,  and  Metcalf,  B.G.  (April  1998).  “Capturing  Eogistics  from  Simulations:  DEPTH 
Technical  Order  Generation.”  United  States  Air  Eorce  Research  Eaboratory,  Hughes  Missile 
Systems  Company,  144  pgs. 

DEPTH  uses  a  human  figure  model  to  visualize  man-machine  interaction  and  receive  on-line 
human  factors  information  simulations;  such  as  simulating  maintenance  evaluations.  The 
primary  focus  of  the  study  is  to  investigate  how  DEPTH  can  support  the  automated  generation  of 
technical  manuals  in  order  to  streamline  the  USAE  technical  order  general  process.  DEPTH  can 
produce  text  describing  maintenance  task  steps  suitable  for  Interactive  Electronic  Technical 
Manuals  (lETMs).  The  language  initially  produced  will  consist  of  simple  instructions,  by  using 
commercial  lETM  authoring  tools. 


[212]  Ei  Pi  Su,  D.  (1997).  “Application  of  New  Information  Technology  to  DOD  Eegacy  Paper 
Technical  Manuals.”  IEEE:  18-22. 

The  Department  of  Defense  is  in  the  process  of  digitizing  paper  technical  manuals  (TMs).  This 
consists,  mainly,  of  direct  copy  to  disk,  with  only  a  few  being  upgraded  to  Electronic  Technical 
Manual  (ETM)  or  Interactive  Electronic  Technical  Manual  (lETM).  However,  there  is  no  clear- 
cut  methodology  for  cost-effectively  converting  paper  manual  to  various  levels  of  upgrade.  The 
Advanced  Technology  Office,  the  U.S.  Army  Test,  Measurement,  and  Diagnostics  Equipment 
Activity,  the  U.S.  Army  Missile  Command,  has  investigated  the  conversion  technologies/tools 
and  developed  conversion  procedures  that  can  achieve  these  goals:  the  most  cost  effective,  value- 
added,  and  user-friendly  ETM/IETM.  This  article  will  provide  some  guidelines  to  assist  the 
ETMs/IETMs  developers  to  achieve  these  goals. 

[213]  McMahon,  P.T.,  McDaniel,  N.A.,  and  Riffee,  J.  (April  2000).  “Interactive  Electronic  Training 
Manual  (lETM)  Guide,  Eirst  Edition.”  Defense  Systems  Management  College  Press,  Et.  Belvoir, 
VA. 

The  rapid  integration  of  Interactive  Electronic  Technical  Manuals  (lETMs)  into  military 
workspaces  throughout  the  Department  of  Defense  has  created  an  opportunity  to  enhance  the 
comprehensive  instruction  provided  to  students  of  Defense  Acquisition  University  (DAU) 
courses  and  to  members  of  the  defense  acquisition  workforce.  Designed  for  use  in  DAU  courses 
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and  as  an  aid  to  acquisition  managers,  the  text  initially  assumes  the  reader  has  little  familiarity 
with  lETMs.  It  presents  fundamental  material,  and  then  addresses  issues  of  immediate  concern  to 
the  acquisition  managers  from  both  the  government  and  industry  perspective.  This  document  is 
designed  to  be  the  primary  desk  reference  for  acquisition  personnel  who  will  be  required  to 
acquire,  develop,  deliver  and/or  manage  lETMs.  It  incorporates  the  status  of  existing/planned 
DoD  and  Service-unique  policy  guidance;  discusses  current  and  projected  technologies  related  to 
the  production  of  lETMs;  analyzes  the  relationship  between  lETMs  and  training;  and  addresses 
delivery  vehicles  -  including  the  World  Wide  Web  (WWW). 


[214]  Navy,  Department  of  the.  (2000).  Training/Interactive  Electronic  Technical  Manual  Interface 
Guide,  Department  of  the  Navy,  USA:  19 1  pgs. 

This  Training/Interactive  Electronic  Technical  Manual  Interface  Guide  (TUG)  is  designed  to:  (I) 
Support  the  general  lETM  acquisition  guidelines  in  the  lETM  Process  Plan  with  more  specific 
requirements  for  interfaces  between  lETMs  and  training  development  and  delivery  tools,  (2) 
Present  a  business  case  structure  for  an  interface  between  lETMs  and  training  tools,  (3)  Supply 
business  case  analysis  tools  for  Acquisition/Eogistics  Managers  to  assess  the  feasibility  of 
expanding  lETM  functionality  to  support  interfaces  with  training  tools,  and  (4)  Provide  a 
checklist  for  those  managers  to  define  development  and  implementation  of  lETM  functionality 
required  to  support  the  training  interface. 


[215]  Rivera,  G.  A.  (1996).  Human  Eactor  Issues  in  Interactive  Electronic  Technical  Manuals  for 
Aircraft  Maintenance. 

http  ://members  .aol.com/geo  1 3/ietm.htm 

Advances  in  computer  technology  and  Compact  Disc-Read  Only  Memory  (CD-ROM)  have 
given  way  to  the  advent  of  the  Interactive  Electronic  Technical  Manual  (lETM).  lETMs  have 
compressed  the  numerous  volumes  of  text  into  just  a  hand  full  of  CD-ROMs  loaded  with  the 
text,  sound  and  video.  This  brings  into  play  new  human  factors  issues.  What  is  the  proper  and 
most  comprehensive  way  to  display  this  information?  Is  it  safe  to  assume  that  the  user  can  use 
the  software  efficiently?  Can  an  electronic  system  be  reliable  enough  to  convey  the  information 
without  being  confusing  or  misleading?  This  paper  will  try  to  answer  these  questions  and  address 
other  problems  faced,  with  the  introduction  of  lETMs. 

[216]  Scholtes,  R.  (April  17,  1999).  “Interactive  Electronic  Technical  Manuals:  An  End-Users’ 
Perspective.”  Thesis,  Department  of  Industrial  Technology,  Eastern  Michigan  University, 
Ypsilanti,  Michigan. 

The  Department  of  Defense  is  converting  its  paper  technical  manuals  to  an  interactive  electronic 
format  as  a  cost  cutting  measure.  One  of  the  first  technical  manuals  to  be  converted  to  an 
electronic  media  by  the  U.S.  Army  is  for  the  Palletized  Eoad  System.  This  research  focused  on 
evaluating  the  interactive  electronic  technical  manual  (lETM)  for  the  Palletized  Eoad  System 
from  an  end-users’  perspective.  Input  from  the  end-user  is  required  to  ensure  that  the  usability  of 
these  technical  manuals  is  not  compromised.  End-users’  expectations  were  gathered  through 
focus  groups  and  questionnaires  at  six  research  sites.  Research  findings  included  the  need  to  (a) 
provide  provisions  to  update  lETM  information,  (b)  provide  initial  comprehensive  lETM  training 
with  periodic  on-site  training  updates,  (c)  improve  diagnostic  and  troubleshooting  capability,  (d) 
provide  color  display,  printing,  data  storage  and  retrieval  capability,  (e)  provide  procedures  to 
maintain  security,  and  (f)  reduce  the  size  and  weight  of  the  prototype  lETM. 
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[217]  Venntronix  Interactive  Electronic  Technical  Manual  (lETM)  Training  Tutorial.  (Eebruary 

2000). 

http://lrc3.monmouth.armv.mil/cecom/lrc/pie/etm/ietm  1  .html 

Training  Contents  consist  of:  Module  1 -Introduction  and  Background  (Purpose,  Scope, 
Assumptions,  Terms  and  acronyms.  Background,  and  CECOM  Policy),  and  Module  2- 
ETM/IETM  Eearning/Refresher  Module  (ETMs,  lETMs,  Platforms,  Operating  System,  and 
Authoring  Systems). 


[218]  Zachary,  W.  Cannon-Bowers,  J.  Bilazarian,  P.  Krecker,  D.  Eardieri,  P.  and  Bums,  J.  (1999). 
“The  Advanced  Embedded  Training  System  (AETS):  An  Intelligent  Embedded  Tutoring  System 
for  Tactical  Team  Training.”  Journal  of  Artificial  Intelligence  in  Education:  25  pgs. 

The  Advanced  Embedded  Training  System  (AETS)  applies  intelligent  tutoring  systems 
technology  to  improving  tactical  training  quality  and  reducing  manpower  needs  in  simulation- 
based  shipboard  team  training.  AETS  provides  layers  of  performance  assessment,  cognitive 
diagnosis,  and  team-training  support  on  top  of  the  existing  embedded  mission  simulation 
capability  in  the  Navy’s  Aegis-class  ships.  Detailed  cognitive  models  of  trainee  task  performance 
are  used  to  drive  the  assessment,  diagnosis  and  instructional  functions  of  the  system.  AETS’  goal 
is  not  to  replace  human  instructors,  but  to  allow  one  instructor  to  perform  the  work  of  several, 
and  in  a  more  consistent  and  efficient  manner  than  possible  today. 
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3.  STANDARDS 

a.  Government 

[219]  - (1995).  MIL- PRF- 87269  Military  Specification,  Database,  Revisable:  Interactive 

Electronic  Technical  Manuals,  for  the  Support  of. 

[220]  - (1995).  MIL-PRF-87268  Implementation  Guide,  Manuals,  Interactive  Electronic 

Technical:  General  Content,  Style,  Format,  and  User-Interaction  Requirements. 

[221]  - MIE-M-81929  Manuals,  Technical:  Illustrated  Parts  Breakdown,  (Work  Package 

Concept);  Preparation  of. 

[222]  - MIE-M-81928  Manuals,  Technical:  Aircraft  and  Aeronautical  Equipment 

Maintenance,  Preparation  of  (Work  Package  Concept). 

[223]  - MIE-M-81927  Manuals,  Technical;  General  Style  and  Format  of  (Work  Package 

Concept). 

[224]  - MIE-M-38784C  Manuals,  Technical;  General  Style  and  Format  Requirements. 

[225]  - (1996).  MIF-PRF-29612  Performance  Specification  Training  Data  Products. 

[226]  - (1992).  MIF-PRF-28003A  Digital  Representation  for  Communication  of  Illustration 

Data:  CGM  Application  Profile. 

[227]  - (1992).  MIF-PRF-28002  Raster  Graphics  Representation  In  Binary  Format, 

Requirements  For. 

[228]  - (1991).  MIF-PRF-28001  Markup  Requirements  and  Generic  Style  Specification  for 

Electronic  Printed  Output  and  Exchange  of  text. 

[229]  - (1992).  MIF-PRF-28000A  Amendment  1,  Digital  Representation  for  Communication 

of  Product  Data:  IGES  Application  Subsets  and  IGES  Application  Protocols. 
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[230]  - (1993).  OPNAV  Instruction  5230.24  Navy  and  Marine  Corps  Policy  on  the  Use  of 

Compact  Disc  Technology. 

[231]  - (1992).  MIL-STD-1840C  Automated  Interchange  of  Technical  Information. 

[232]  - (1990).  MIL-STD-1808  System/Subsystem/Subject  Number  (S/S/SN)  Numbering 

System. 

[233]  - (1997).  MIL-HDBK-1379-4  Department  of  Defense  Handbook:  Glossary  for  Training. 

[234]  - (1997).  MIL-HDBK-1379-3  Department  of  Defense  Handbook:  Development  of 

Interactive  Multimedia  Instruction  (IMI). 

[235]  - (1997).  MIL-HDBK-1379-1  Department  of  Defense  Handbook:  Guidance  for 

Acquisition  of  Training  Data  Products  and  Services. 

[236]  - NAVEDTRA  131  Personnel  Performance  Profile  Based  Curriculum  Development 

Manual. 

[237]  - NAVEDTRA  130  Task  Based  Curriculum  Development  Manual. 

[238]  - (1995).  S0005-AD-PRO-010  Navy  lETM  Process  Plan. 

[239]  - CAES  lETM  Web  Site.  (2000). 

http://navvsgml.dt.navv.mil/ietm/ietm.html 

[240]  - (1998).  lETM  Specification  Development. 

http  ://w  ww  .dsmc  .dsm.mi1/r/port/ff/ful3287d.htm 

[241]  - Welcome  to  the  NAVAIR  lETM  Standardization  Demonstration.  (1997). 

http://venus.dcnicn.com/antech/CAES97/index.htm 

This  demonstration  describes  an  lETM  Architecture  being  developed  for  NAVAIR,  and  also 
illustrates  many  of  the  key  concepts  in  action.  The  architecture  is  under  consideration  by  the  Tri- 
Services  lETM  Technology  Working  Group  for  application  across  DoD.  The  benefits  of  the 
architecture  are  primarily  to  end  users  of  lETM  systems,  with  additional  advantages  to  authors 
and  developers.  End  users  will  see  a  variety  of  lETM  sources  combined  in  single,  integrated 
presentations  using  technology  developed  for  the  Web. 
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[242] - (1995).  NAMTRAGRU  Memphis  Computer  Based  Training  Conventions  and 

Standards. 


[243]  NAWC-AD  (1995).  Metafile  for  Interactive  Documents,  MID  Application  Guide.  Patuxent 
River,  MD. 


[244] - (1993).  Applying  CALS  to  the  Creation,  Management  and  Use  of  Technical  Manuals, 

2"‘^  Edition  (CALS  Resource  and  Implementation  Cooperation). 


[245]  - (1992).  Navy  Interactive  Courseware  (ICW)  Style  Guide. 

[246]  - (1992).  Quality  Assurance  Program:  Interactive  Electronic  Technical  Manuals  and 


Associated  Technical  Information;  Requirements  for  (MIL-Q-87270). 

This  Specification  prescribes  the  requirements  for  a  contractor’s  Quality  Assurance  (QA) 
program  for  Interactive  Electronic  Technical  Manuals  (lETMs)  and,  where  procured,  the 
associated  lETM  data  base.  The  requirements  herein  cover  the  QA  process  from  planning 
through  final  submission  of  the  delivered  product  for  acceptance;  and  apply  as  well  to  changes 
and  revisions  thereto. 


[247] - Computer-Aided  Acquisition  and  Logistic  Support  Implementation  Plan  (CALS IP). 


[248]  Gallager,  S.  (1995).  “Navy  Class  is  All  Electronic.”  Government  Computer  News  14(7):  40. 


[249]  Kribs,  H.  D.  Mark,  L.J.  Morris,  B.A.  and  Dickason,  D.K.  (1996).  A  Survey  of  Interactive 

Electronic  Technical  Manuals  Used  in  Education  and  Training  (NPRDC  TN-96-35).  San  Diego, 
Navy  Personnel  Research  and  Development  Center. 


[250]  Mark,  L.  J.  Kribs,  H.D.  Morris,  B.A.  and  Dickason,  D.K.  (1996).  Cost/Benefits  Analysis  of 
Interactive  Electronic  Technical  Manuals/ Automated  Classroom  (lETM/AC)  Technology 
(NPRDC  TN-96-34).  San  Diego,  Navy  Personnel  Research  and  Development  Center. 


[251]  Memorandum,  Assistant  Secretary  of  the  Navy  (RD&A)  (1999).  Department  of  the  Navy  Policy 
on  Digital  Logistics  Technical  Data. 


[252]  Morris,  B.  A.  and  Dickason,  D.K.  (1996).  Evaluation  of  the  Interactive  Electronic  Technical 
Manual/ Automated  Classroom  (lETM/AC)  (NPRDC  TN-96-39).  San  Diego,  Navy  Personnel 
Research  and  Development  Center. 


©  Copyright  2002,  Carnegie  Mellon  University 
October  2002 


73 


b.  ISO 


[253] - (1999).  Papers  on  HyTime,  HyTime  Users’  Group.  1999. 

www.hvtime.org/papers/index.html 


[254]  SGML  User’s  Group.  (1998).  HyTime  Users’  Group  Home  Page  (ISO/IEC  10744:1992), 
SGML  Users’  Group.  1998. 
www.hvtime.org 


[255] - (1997).  ISO  10744:  Hypermedia/Time -based  Structuring  Language  (Hytime). 


[256]  N1920.  (1997).  Information  Technology  -  Hypermedia/Time-based  Structuring  Language 
(HyTime)  (ISO/IEC  JTC1/SC18AVG8  N1920).  1997. 
www.oml.gOv/sgml/wg8/docs/nl920/html/nl920.html 


[257] - (1996).  Interactive  Electronic  Technical  Manual  (lETM)  Process  Plan  (S0005-AD- 

PRO-010). 


[258] - (1994).  Technical  Resources  for  Developers,  Microsoft  Developer  Network  - 

Development  Eibrary,  CD-ROM  Disc  Seven. 


[259] - SOME  Assessment:  SOME  Acronyms  Explained. 

http://www.ex.ac.uk/SGME/report/annex-c.html 


[260]  Clark,  J.  SP:  An  SGME  System  Conforming  to  International  Standard  ISO  8879  —  Standard 
Generalized  Markup  Eanguage. 

www.iclark.com/sp/ 

[261]  Goldfarb,  C.  E.  (1999).  Project  Editor’s  Review  of  ISO  8879. 
www.sgmlsource.com/goldfarb/8879rev/ 

ISO  8879,  the  SGME  International  Standard,  is  developed  and  maintained  by  ISO/IEC 
JTC1/SC34/WG1;  that  is.  Working  Group  1  of  Subcommittee  34  of  Joint  Technical  Committee 
of  the  International  Organization  for  Standardization  (ISO)  and  the  International  Electrotechnical 
Commission. 


[262]  Goldfarb,  C.  E.  (1998).  ISO/IEC  JTC1/SC34  Document  Description  and  Processing  Eanguages. 
1998. 

w  w  w .  s  gmlsource .  com/go  ldfarb/8  87 9rev/n0029 .htm 

This  annex  remedies  defects  revealed  by  the  multiple  adaptations  of  SGME  for  the  World  Wide 
Web  (WWW),  intranets,  and  extranets.  The  annex  corrects  errors,  resolves  ambiguities  for  which 
there  is  a  clear  resolution  that  does  not  cause  existing  conforming  documents  to  become  non- 


©  Copyright  2002,  Carnegie  Mellon  University 
October  2002 


74 


conforming,  and  provides  a  choice  of  alternative  resolutions  for  other  ambiguities.  Although 
motivated  by  the  World  Wide  Web,  applicability  of  this  annex  extends  to  all  uses  of  SGML. 


[263]  Goldfarb,  C.  F.  (1997).  SGML  Extended  Facilities.  1997. 
www.sgmlsource.com/goldfarb/8879rev/sgmlxfac.htm 


[264]  Goldfarb,  C.  F.  (1996).  N1896  Rev  ISO/IEC  JTC1/SC18/WG8  Document  Processing  and 
Relating  Communication  -  Document  Description  and  Processing  Fanguages. 
www.sgmlsource.com/goldfarb/8879rev.nl896rev.htm 

This  technical  communication  adds  a  brief  annex  to  ISO  8879  to  meet  an  urgent  need  for 
extended  naming  rules  for  non-Fatin  scripts  in  support  of  the  following  statements  in  clause  0.2: 
there  must  be  no  national  language  bias,  and  the  characters  used  for  names  can  be  augmented  by 
any  special  national  characters. 


[265]  Goldfarb,  C.  F.  (1996).  Third  Interim  Report  on  the  Project  Editor’s  Review  of  ISO  8879.  1996. 
www.sgmlsource.com/goldfarb/8879rev/nl855.htm 

WG8  has  directed  the  Project  Editor  of  ISO  8879  (SOME)  to  conduct  a  systematic  review  of  the 
standard  to  consider  future  development.  As  the  review  is  in  its  final  stages  and  participation  is 
increasing,  this  report  incorporates  the  substance  of  both  previous  reports  (N1605  and  N1701) 
and  some  other  documents  in  order  to  reduce  the  number  of  documents  that  need  be  accessed  by 
new  participants. 


[266]  Goldfarb,  C.  F.  ISO/IEC  JTCI/SC18/WG8/NI035:  Recommendations  for  a  Possible  Revision 
of  ISO  8879. 

w  w  w .  s  gmlsource .  com/go  ldfarb/8  87 9rev/n  1035.  htm 


[267]  Goldfarb,  C.  Newcomb,  S.R.  Kimber,  W.E.  and  Newcomb,  P.J.  (1997).  A  Reader’s  Guide  to  the 
HyTime  Standard  (ISO/IEC  10744),  The  HyTime  Users’  Group.  1997. 
www.hvtime.org/papers/htguide.html 


[268]  Goldfarb,  C.  Newcomb,  S.R.  Kimber,  W.E.  and  Newcomb,  P.J.  (1997).  Information 
processing — Hypermedia/Time-based  Structured  Eanguage  (HyTime)  -  2"‘^  Edition  (ISO/IEC 
10744:1997). 

www.oml.gov/sgml/wg8/docs/nl920/ 

[269]  Kimber,  W.  E.  (1999).  A  Tutorial  Introduction  to  SOME  Architectures.  1999. 
www.isogen.com/papers/archintro.html 

Provides  a  brief,  tutorial  introduction  to  SGME  architectures. 


[270]  Kimber,  W.  E.  (1997).  What’s  New  and  Cool  in  HyTime  (ISO/IEC  10744:1997  HyTime, 
Second  Edition)  1997. 
www.isogen.com/papers/newcool.html 


©  Copyright  2002,  Carnegie  Mellon  University 
October  2002 


75 


[271]  Leggott,  M.  (1996).  Hypermedia  and  the  Net.  1996. 

[272]  van  Herwijnen,  E.  (1994).  Practical  SGML,  2”^^  Edition.  Kluwer  Academic  Publishers. 


©  Copyright  2002,  Carnegie  Mellon  University 
October  2002 


76 


